- slurm-users - lists.schedmd.com

Slurm hanging behavior
by Richard Yang 20 Aug '24

20 Aug '24

Hello Slurm community, We are using slurm as the system to deploy training jobs on a large gpu cluster, but encounter a strange behavior. As new comers, we wonder if this is a known behavior. Below is some more info: * We are running a relatively older version 22.0.5 * At relatively higher load, we encountered hanging. It is particularly puzzling in the following sense: assume we have nodelist1 with 6 hosts and nodelist2 with 7 hosts. We run simple ‘hostname’. Deploying on nodelist1 alone or nodelusr2 alone will be fine, but with all 13 hosts, the debug messages show that the execution hang after showing that the last task done. It then hangs for exactly 180 seconds. Does anyone know the potential issue? We sure be happy to post more config details or debug messages. Thank you so much! Richard

1 0

Access to --constraint=<arg> in Lua cli_filter?
by Kevin Buckley 20 Aug '24

20 Aug '24

If I supply a --constraint=<arg> option to an sbatch/salloc/srun, does the arg appear inside any object that a Lua CLI Filter could access? I've tried this basic check if is_unset(options['constraint']) then slurm_errorf('constraint is unset ') end and seen that that object is, indeed, unset. Kevin

2 2

Unable to run sequential jobs simultaneously on the same node
by Arko Roy 20 Aug '24

20 Aug '24

I want to run 50 sequential jobs (essentially 50 copies of the same code with different input parameters) on a particular node. However, as soon as one of the jobs gets executed, the other 49 jobs get killed immediately with exit code 9. The jobs are not interacting and are strictly parallel. However, if the 50 jobs run on 50 different nodes, it runs successfully. Can anyone please help with possible fixes? I see a discussion almost along the similar lines in https://groups.google.com/g/slurm-users/c/I1T6GWcLjt4 But could not get the final solution. -- Arko Roy Assistant Professor School of Physical Sciences Indian Institute of Technology Mandi Kamand, Mandi Himachal Pradesh - 175 005, India Email: arko(a)iitmandi.ac.in Web: https://faculty.iitmandi.ac.in/~arko/

5 7

sreport syntax for TRES/GPU usage
by Robert Kudyba 16 Aug '24

16 Aug '24

In a 25 node heterogeneous cluster with 4 different types of GPUs, to get granular to see which GPUs were used most over a time period we have to set AccountingStorageTRES to something like: AccountingStorageTRES=gres/gpu,gres/gpu:rtx8000,gres/gpu:v100s,gres/gpu:a40,gres/gpu:a100 Unfortunately it's currently at: AccountingStorageTRES=gres/gpu At least all nodes have the same GPU within each node. What are some good options to sreport to get details on usage over a year, e.g., percentage of CPU vs GPU, which partitions/accounts used the most GPUs, etc. From this example: sreport -tminper -t Percent cluster utilization --tres="cpu,gres/gpu" start=2023-07-01 -------------------------------------------------------------------------------- Cluster Utilization 2023-07-01T00:00:00 - 2024-08-15T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Reserved Reported --------- -------------- ----------- ---------- -------- ----------- ---------- ----------- cluster cpu 43.81% 2.87% 0.00% 48.35% 4.97% 99.86% cluster gres/gpu 50.36% 3.59% 0.00% 46.05% 0.00% 100.38% Is that showing that 50% of all jobs were run with GPUs? How do we read the Idle column? Why does Reported show > 100% for gres?

1 0

Upgrade compute node to 24.05.2
by Sid Young 15 Aug '24

15 Aug '24

G'Day all, I've been upgrading cmy cluster from 20.11.0 in small steps to get to 24.05.2. Currently 1 have all nodes on 23.02.8, the controller on 24.05.2 and a single test node on 24.05.2. All are Centos 7.9 (upgrade to Oracle Linux 8.10 is Phase 2 of the upgrades). When I check the slurmd status on the test node I get: [root@hpc-dev-01 24.05.2]# systemctl status slurmd ● slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2024-08-15 10:45:15 AEST; 24s ago Main PID: 46391 (slurmd) Tasks: 1 Memory: 1.2M CGroup: /system.slice/slurmd.service └─46391 /usr/sbin/slurmd --systemd Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Considering each NUMA node as a socket Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Node reconfigured socket/core boundaries SocketsPerBoard=4:8(hw) CoresPerSocket=16:8(hw) Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Considering each NUMA node as a socket Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: slurmd version 24.05.2 started Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: *plugin_load_from_file: Incompatible Slurm plugin /usr/lib64/slurm/mpi_none.so version (23.02.8)* Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: error: Couldn't load specified plugin name for mpi/none: Incompatible plugin version Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: error: MPI: Cannot create context for mpi/none Aug 15 10:45:15 hpc-dev-01 systemd[1]: Started Slurm node daemon. Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: slurmd started on Thu, 15 Aug 2024 10:45:15 +1000 Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: CPUs=64 Boards=1 Sockets=8 Cores=8 Threads=1 Memory=257778 TmpDisk=15998 Uptime=2898769 CPUSpecL...ve=(null) Hint: Some lines were ellipsized, use -l to show in full. [root@hpc-dev-01 24.05.2]# We don't use MPI (life science workloads)... should I remove the file? If it is version 23.02.8 then doesn't 24.05.2 have that plugin built in? There are no references to mpi i the slurm.conf file Sid

2 1

Seeking Commercial SLURM Subscription Provider
by John Joseph 14 Aug '24

14 Aug '24

Dear All, Good morning. We successfully implemented a 4-node SLURM cluster with shared storage using GlusterFS and were able to run COMSOL programs on it. After this learning experience, we've determined that it would be beneficial to switch to a commercial SLURM subscription for better support. We are currently seeking a solution provider who can offer support based on a commercial subscription. I would like to reach out to the group for recommendations or advice on how we can avail these services commercially. Thank you.Joseph John

2 1

Annoying canonical question about converting SLURM_JOB_NODELIST to a host list for mpirun
by Jeffrey Layton 12 Aug '24

12 Aug '24

Good afternoon, I know this question has been asked a million times, but what is the canonical way to convert the list of nodes for a job that is container in a Slurm variable, I use SLURM_JOB_NODELIST, to a host list appropriate for mpirun in OpenMPI (perhaps MPICH as well)? Before anyone says, compile OpenMPI with Slurm, I can't change the Slurm installation. I have a script that does the conversion on a single node, but when I try a cluster that does not include the single node, I get an error: scontrol: error: host list is empty The line in the script corresponding to this is, list=$(scontrol show hostname $SLURM_NODELIST) I've tried using the env variable SLURM_JOB_NODELIST and I get the same error message. Thanks! Jeff

3 8

FairShare if there's only one account?
by Drucker, Daniel 10 Aug '24

10 Aug '24

Simple question: Does FairShare still work if every user is under one account? E.g.: $ sacctmgr show assoc format=Account,User Account User ---------- ---------- root root root mic mic asmith mic bsmith mic csmith mic djones mic ejones mic frubble Will it divide time up fairly between the users? I have: PriorityType=priority/multifactor PriorityFavorSmall=YES PriorityWeightAge=50000 PriorityWeightFairshare=100000 PriorityWeightJobSize=0 PriorityWeightQOS=0 In 21.08.8. -- Daniel M. Drucker, Ph.D. Director of IT, MGB Imaging at Belmont McLean Hospital, a Harvard Medical School Affiliate The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

5 31

Jobs distribution over CPUs
by Rafał Lalik 09 Aug '24

09 Aug '24

Hi, I have a very simple computing farm on a single PC with AMD Ryzen 7950X (2x16 cores). I have configured my slurm to use up to 25 CPUs: NodeName=palmer CPUs=25 RealMemory=40000 State=UNKNOWN # Boards=1 SocketsPerBoard=1 CoresPerSocket=16 ThreadsPerCore=2 PartitionName=main Nodes=ALL Default=YES MaxTime=INFINITE State=UP But using htop I see that with all 25 jobs running I use max 16 cores. It seems that like * 6 jobs are using 100% of CPU * 20 jobs use 50% CPU each I use slurm 24.05.2, but I am pretty sure in the past when using I think one of 22.x version, the jobs distribution was like 25 CPUs for 25 jobs with 100% each. Is there anything I can do in the configuration of my farm to have better utilisation of CPU cores? Right now like half of them is not really used. Regards, Rafał

1 0

The issue in the distribution of job
by Sundaram Kumaran 09 Aug '24

09 Aug '24

Dear All, May I have your suggestion in my issue facing, While the job is launched using "salloc -N4--mem 4000 -p active" I find the job is running in the one compute node and the other 3 machines are free, I don`t find the job is distributed evenly, May I have your suggestion, I do squeue /scontrol to find the job distribution and it displays the 4 machines but when I check on the respective machines I don`t find the job running only one machine takes the whole node, Is there any issue in my conf file or what needs to be done, May I have your suggestion pls. FYI, salloc -N4 --mem 4000 -p active [cid:image001.png@01DAE8B4.215FBA50] While using TOP, I find only Debussy is used heavily, I don`t find my job is evenly distributed, May I have your guidance pls. [cid:image002.jpg@01DAEA98.2573DE10] Regards, KumaranS This e-mail and any attachments are only for the use of the intended recipient and may contain material that is confidential, privileged and/or protected by the Official Secrets Act. If you are not the intended recipient, please delete it or notify the sender immediately. Please do not copy or use it for any purpose or disclose the contents to any other person.

2 1

Optimizing CPU socket affinities and NVLink
by Matthew R. Baney 08 Aug '24

08 Aug '24

Hello, I've recently adopted setting AutoDetect=nvml in our GPU nodes' gres.conf files to automatically populate Cores and Links for GPUs, which has been working well. I'm now wondering if I can prioritize having single GPU jobs scheduled on NVLink pairs (these are PCIe A6000s) where one of the GPUs in the pair is already running a single GPU job, assuming the CPU socket with affinity has enough cores to handle the job. We have some users wanting to run single GPU jobs and others wanting to run dual GPU jobs, both on the same nodes, so we would prefer not to configure each NVLink pair as a single GRES, for better job throughput. As is, I've observed that for a node with at least 4 GPUs and 2 sockets (one NVLink pair per socket), Slurm will prioritize evening out core allocation between the sockets. Once the second single GPU job is submitted, one GPU in each NVLink pair is taken up and a subsequent dual GPU job can still run, but doesn't have access to an NVLink pair. We've also got a few nodes where single GPUs have failed, resulting in some NVLink'd pairs and usually a single non-NVLink'd GPU (3 or 7 total GPUs). It'd be ideal if single GPU jobs also got prioritized for scheduling on the non-NVLink'd GPU in this case. Is this possible? All the best, Matthew -- Matthew Baney Assistant Director of Computational Systems mbaney(a)umd.edu | (301) 405-6756 University of Maryland Institute for Advanced Computer Studies 3154 Brendan Iribe Center 8125 Paint Branch Dr. College Park, MD 20742

1 0

Integrate podman to run containers
by amin.raeiszadeh＠gmail.com 08 Aug '24

08 Aug '24

Hello, I have a test cluster consist of two nodes, one as controller and the other as compute node. I followed all the steps from slurm documentation and I want to run jobs as containers but I get the following error when running podman run hello-world on controller node: time="2024-08-06T12:02:54+02:00" level=warning msg="freezer not supported: openat2 /sys/fs/cgroup/system.slice/slurmstepd.scope/job_332/step_0/user/arlvm6.ara.332.0.0/cgroup.freeze: no such file or directory" srun: error: arlvm6: task 0: Exited with exit code 1 time="2024-08-06T12:02:54+02:00" level=warning msg="lstat /sys/fs/cgroup/system.slice/slurmstepd.scope/job_332/step_0/user/arlvm6.ara.332.0.0: no such file or directory" time="2024-08-06T12:02:54+02:00" level=error msg="runc run failed: unable to start container process: unable to apply cgroup configuration: rootless needs no limits + no cgrouppath when no permission is granted for cgroups: mkdir /sys/fs/cgroup/system.slice/slurmstepd.scope/job_332/step_0/user/arlvm6.ara.332.0.0: permission denied" As I tracked on the compute node this path exists /sys/fs/cgroup/system.slice/slurmstepd.scope/ but it looks that could not create the job_332/step_0/user/arlvm6.ara.332.0.0 . The cgroup.conf: CgroupPlugin=cgroup/v2 ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes It is the oci.conf: EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" RunTimeQuery="runc --rootless=true --root=/run/user/1223609544/ state %n.%u.%j.%s.%t" RunTimeKill="runc --rootless=true --root=/run/user/1223609544/ kill -a %n.%u.%j.%s.%t SIGKILL" RunTimeDelete="runc --rootless=true --root=/run/user/1223609544/ delete --force %n.%u.%j.%s.%t" RunTimeRun="runc --rootless=true --root=/run/user/1223609544/ run %n.%u.%j.%s.%t -b %b" As you see I changed the kill command a bit because without SIGKILL param it could not kill the containers. I test again the oci run time on both controller and compute nodes and I think might be helpful to mention two points: the delete command will not work because if you kill the container then there is no resource to be deleted at least in my tests. there is no pause and resume in oci.conf but I test them and got the same error for freezer support and cgroup permissions.

1 0

Find out submit host of past job?
by Steffen Grunewald 07 Aug '24

07 Aug '24

Hello everyone, I've grepped the manual pages and crawled the 'net, but couldn't find any answer to the following problem: I can see that the ctld keeps a record of it below /var/spool/slurm - as long as the job is running or waiting (and shown by "squeue") - and that this stores the environment that contains SLURM_SUBMIT_HOST - but this information seems to be lost when the job finishes. Is there a way to find out what the value of SLURM_SUBMIT_HOST was? I'd be interested in a few more env variables, but this one should be sufficient for a start... Is "sacct" just lacking a job field, or is this info indeed dropped and not stored in the DB? Thanks, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

6 9

"error: slurm_auth_get_host: Lookup failed: Unknown host" in slurmctld.log
by Chris Taylor 07 Aug '24

07 Aug '24

I keep getting this logged on my Slurm control host: [2024-08-06T20:31:40.196] error: slurm_auth_get_host: Lookup failed: Unknown host I don't see an identifiable pattern and I'm not sure how to troubleshoot. The jobs being submitted at or around that time seem fine and nobody's complained about their jobs not running, so I can't track down a bad or nonexistent DNS record. Chris

1 0

With slurm, how to allocate a whole node for a single multi-threaded process?
by Henrique Almeida 06 Aug '24

06 Aug '24

Hello, everyone, with slurm, how to allocate a whole node for a single multi-threaded process? https://stackoverflow.com/questions/78818547/with-slurm-how-to-allocate-a-w… -- Henrique Dante de Almeida hdante(a)gmail.com

8 16

LRMS error: (-1) Job missing from SLURM."
by Felix 06 Aug '24

06 Aug '24

Hello at site RO-14-ITIM, after a power failure I get the following problem 2024-08-06 15:53:04 Finished - job id: c9INDmclYv5ngvuSSqSAreymYz3jwmOETUEmV71LDmABFKDm7KNpMn, unix user: 1900:1900, name: "org.nordugrid.ARC-CE-result-ops", owner: "/dc=eu/dc=egi/c=hr/o=robots/o=srce/cn=robot:argo-egi@cro-ngi.hr", lrms: SLURM, queue: debug, lrmsid: 274399, failure: "LRMS error: (-1) Job missing from SLURM." 2024-08-06 15:53:04 Finished - job id: tjJNDmclYv5ngvuSSqSAreymYz3jwmOETUEmd71LDmABFKDmePf7To, unix user: 1900:1900, name: "org.nordugrid.ARC-CE-result-ops", owner: "/dc=eu/dc=egi/c=hr/o=robots/o=srce/cn=robot:argo-egi@cro-ngi.hr", lrms: SLURM, queue: debug, lrmsid: 274400, failure: "LRMS error: (-1) Job missing from SLURM." 2024-08-06 15:53:04 Finished - job id: kiJNDmclYv5ngvuSSqSAreymYz3jwmOETUEml71LDmABFKDmCmwifm, unix user: 1900:1900, name: "org.nordugrid.ARC-CE-result-ops", owner: "/dc=eu/dc=egi/c=hr/o=robots/o=srce/cn=robot:argo-egi@cro-ngi.hr", lrms: SLURM, queue: debug, lrmsid: 274398, failure: "LRMS error: (-1) Job missing from SLURM." The jobs can not be seen in sinfo or squeue And indication on how where to look up the problem? Thank you Felix -- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

2 1

when running `salloc --gres=gpu:1` should I see all gpus in nvidia-smi ?
by Oren 05 Aug '24

05 Aug '24

Hello, When I am running this command: `salloc --nodelist=gpu03 -p A4500_Features --gres=gpu:1` and then automatically ssh to the job, what should I see when I run nvidia-smi? All the GPUs in the host or just a single one? Thanks

2 2

_refresh_assoc_mgr_qos_list: no new list given back keeping cached one
by joao.damas＠syngenta.com 05 Aug '24

05 Aug '24

Hi all, We are doing a simple setup for a Slurm cluster (version 23.11.6). We follow the documentation and we are trying a setup still without accounting or slurmdbd. The slurm.conf is really simple: ``` ClusterName=Develop SlurmctldHost=head # Slurm configuration AuthType=auth/munge CryptoType=crypto/munge SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdLogFile=/var/log/slurm/slurmd.log SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurmctld # Nodes NodeName=worker1 CoresPerSocket=2 Sockets=1 ThreadsPerCore=1 NodeName=worker2 CoresPerSocket=2 Sockets=1 ThreadsPerCore=1 # Partitions PartitionName=develop Default=YES MaxTime=UNLIMITED Nodes="worker1,worker2" ``` When running a simple `srun sleep 10`, all works well and the log file shows: [2024-05-15T12:34:12.741] sched: _slurm_rpc_allocate_resources JobId=1 NodeList=worker1 usec=549 [2024-05-15T12:34:22.775] _job_complete: JobId=1 WEXITSTATUS 0 [2024-05-15T12:34:22.775] _job_complete: JobId=1 done But when creating a scrip with the same sleep command, and submiting using `sbatch test.sh`, the log shows: [2024-05-15T12:35:39.916] _slurm_rpc_submit_batch_job: JobId=2 InitPrio=1 usec=368 [2024-05-15T12:35:40.000] error: _refresh_assoc_mgr_qos_list: no new list given back keeping cached one. [2024-05-15T12:35:40.000] sched: JobId=2 has invalid account [2024-05-15T12:35:40.145] sched/backfill: _start_job: Started JobId=2 in develop on worker1 [2024-05-15T12:35:50.172] _job_complete: JobId=2 WEXITSTATUS 0 [2024-05-15T12:35:50.172] _job_complete: JobId=2 done We have the same account with the UID and GID, as said in the documentation. Looking at the function that seems to spit out that error (https://github.com/SchedMD/slurm/blob/e9f28ede27795f525e62f998cb2d40931d884…), it appears like there should be some accounting setup? We do not have slurmdbd setup and the documentation states we should test basic functionality before implementing that daemon. Any tips? Thanks in advance. João

4 3

problem with squeue --json with version 24.05.1
by Markus Köberl 05 Aug '24

05 Aug '24

$ squeue --version slurm 24.05.1 $ squeue --json malloc(): invalid size (unsorted) Aborted forcing an older data_parser version works: $ squeue --json=v0.0.40 regards Markus Köberl -- Markus Koeberl Graz University of Technology Signal Processing and Speech Communication Laboratory E-mail: markus.koeberl(a)tugraz.at

4 5

Upgrade node while jobs running
by Sid Young 03 Aug '24

03 Aug '24

G'day all, I've been waiting for node to become idle before upgrading them however some jobs take a long time. If I try to remove all the packages I assume that kills the slurmstep program and with it the job. Sid

5 5

slumrestd 24.05.1: crashes when GET on /slurm/v0.0.41/nodes : unsorted double linked list corrupted
by Josef Dvořáček 01 Aug '24

01 Aug '24

Isn't this failure familiar to anyone? When I ask API endpoint "localhost:6820/slurm/v0.0.41/jobs", slurmrestd segrafults with unsorted double linked list corrupted. Anyone using this API endpoint without segfaulting? I do the get using curl: curl --header X-SLURM-USER-NAME:root --header X-SLURM-USER-TOKEN:$SLURM_JWT -G localhost:6820/slurm/v0.0.41/jobs In comparison, curl --header X-SLURM-USER-NAME:root --header X-SLURM-USER-TOKEN:$SLURM_JWT -G localhost:6820/slurm/v0.0.41/nodes Works well. josef čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: _on_url: [[localhost]:52909] url path: /slurm/v0.0.41/jobs query: (null) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: operations_router: [[localhost]:52909] GET /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: slurmrestd: operations_router: [[localhost]:52909] GET /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:52909] attempting user_name root token authentication pass through čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:52909] attempting user_name root token authentication pass through čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi.json"](0, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi.yaml"](1, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi"](2, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=2 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi","v3"](3, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match shares to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed shares čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","shares"](4, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match reconfigure to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed reconfigure čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","reconfigure"](5, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match diag to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed diag čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","diag"](6, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match ping to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed ping čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","ping"](7, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match licenses to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed licenses čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","licenses"](8, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: method skip for ["slurm","v0.0.41","job","submit"](9, GET != POST) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","job","submit"](9, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: method skip for ["slurm","v0.0.41","job","allocate"](10, GET != POST) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","job","allocate"](10, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match jobs to jobs: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match successful for ["slurm","v0.0.41","jobs"](11, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: operations_router: [[localhost]:52909] found callback handler: (0x0) callback_tag=0 path=/slurm/v0.0.41/jobs parser=data_parser/v0.0.41 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] did not provide a known content type header. Assuming URL encoded. čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _parse_http_accept_entry: found */* with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] accepts */* with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] found accepts */*=application/json with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug3: _resolve_mime: [[localhost]:52909] mime read: application/x-www-form-urlencoded write: application/json čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug3: _call_handler: [[localhost]:52909] BEGIN: calling ctxt handler: 0x7F9C9D294A36[0] for path: /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: wrap_openapi_ctxt_callback: [[localhost]:52909] GET using data_parser/v0.0.41 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x1 from 0x408376 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x408376 from 0x1 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: accounting_storage/slurmdbd: _connect_dbd_conn: Sent PersistInit msg čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x1 from 0x408376 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x408376 from 0x1 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: malloc(): unsorted double linked list corrupted čec 24 14:37:55 slurmserver2.koios.lan systemd[1]: Started Process Core Dump (PID 1502951/UID 0).

2 3

Slurm versions 24.05.2, 23.11.9, and 23.02.8 are now available (security fix for switch plugins)
by Tim Wickberg 31 Jul '24

31 Jul '24

Slurm versions 24.05.2, 23.11.9, and 23.02.8 are now available and include a fix for a recently discovered security issue with the switch plugins. SchedMD customers were informed on July 17th and provided a patch on request; this process is documented in our security policy. [1] For the switch/hpe_slingshot and switch/nvidia_imex plugins, a user could override the isolation between Slingshot VNIs or IMEX channels. If you do not have one of these switch plugins configured, then you are not impacted by this issue. It is unclear what, if any, information could be accessed with access to an unauthorized channel. This disclosure is being made out of an abundance of caution. If you do have one of these plugins enabled, the slurmctld must be restarted before the slurmd daemons to avoid disruption. Downloads are available at https://www.schedmd.com/downloads.php . Release notes follow below. - Tim [1] https://www.schedmd.com/security-policy/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support > * Changes in Slurm 24.05.2 > ========================== > -- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when > more than 10 threads try to get energy at the same time. This prevented > the possibility to get energy from slurmd by any step until slurmd was > restarted, so losing energy accounting metrics in the node. > -- accounting_storage/mysql - Fix issue where new user with wckey did not > have a default wckey sent to the slurmctld. > -- slurmrestd - Prevent slurmrestd segfault when handling the following > endpoints when none of the optional parameters are specified: > 'DELETE /slurm/v0.0.40/jobs' > 'DELETE /slurm/v0.0.41/jobs' > 'GET /slurm/v0.0.40/shares' > 'GET /slurm/v0.0.41/shares' > 'GET /slurmdb/v0.0.40/instance' > 'GET /slurmdb/v0.0.41/instance' > 'GET /slurmdb/v0.0.40/instances' > 'GET /slurmdb/v0.0.41/instances' > 'POST /slurm/v0.0.40/job/{job_id}' > 'POST /slurm/v0.0.41/job/{job_id}' > -- Fix IPMI energy gathering when no IPMIPowerSensors are specified in > acct_gather.conf. This situation resulted in an accounted energy of 0 > for job steps. > -- Fix a minor memory leak in slurmctld when updating a job dependency. > -- scontrol,squeue - Fix regression that caused incorrect values for > multisocket nodes at '.jobs[].job_resources.nodes.allocation' for > 'scontrol show jobs --(json|yaml)' and 'squeue --(json|yaml)'. > -- slurmrestd - Fix regression that caused incorrect values for > multisocket nodes at '.jobs[].job_resources.nodes.allocation' to be dumped > with endpoints: > 'GET /slurm/v0.0.41/job/{job_id}' > 'GET /slurm/v0.0.41/jobs' > -- jobcomp/filetxt - Fix truncation of job record lines > 1024 characters. > -- Fixed regression that prevented compilation on FreeBSD hosts. > -- switch/hpe_slingshot - Drain node on failure to delete CXI services. > -- Fix a performance regression from 23.11.0 in cpu frequency handling when no > CpuFreqDef is defined. > -- Fix one-task-per-sharing not working across multiple nodes. > -- Fix inconsistent number of cpus when creating a reservation using the > TRESPerNode option. > -- data_parser/v0.0.40+ - Fix job state parsing which could break filtering. > -- Prevent cpus-per-task to be modified in jobs where a -c value has been > explicitly specified and the requested memory constraints implicitly > increase the number of CPUs to allocate. > -- slurmrestd - Fix regression where args '-s v0.0.39,dbv0.0.39' and > '-d v0.0.39' would result in 'GET /openapi/v3' not registering as a valid > possible query resulting in 404 errors. > -- slurmrestd - Fix memory leak for dbv0.0.39 jobs query which occurred if the > query parameters specified account, association, cluster, constraints, > format, groups, job_name, partition, qos, reason, reservation, state, users, > or wckey. This affects the following endpoints: > 'GET /slurmdb/v0.0.39/jobs' > -- slurmrestd - In the case the slurmdbd does not respond to a persistent > connection init message, prevent the closed fd from being used, and instead > emit an error or warning depending on if the connection was required. > -- Fix 24.05.0 regression that caused the slurmdbd not to send back an error > message if there is an error initializing a persistent connection. > -- Reduce latency of forwarded x11 packets. > -- Add "curr_dependency" (representing the current dependency of the job) > and "orig_dependency" (representing the original requested dependency of > the job) fields to the job record in job_submit.lua (for job update) and > jobcomp.lua. > -- Fix potential segfault of slurmctld configured with > SlurmctldParameters=enable_rpc_queue from happening on reconfigure. > -- Fix potential segfault of slurmctld on its shutdown when rate limitting > is enabled. > -- slurmrestd - Fix missing job environment for SLURM_JOB_NAME, > SLURM_OPEN_MODE, SLURM_JOB_DEPENDENCY, SLURM_PROFILE, SLURM_ACCTG_FREQ, > SLURM_NETWORK and SLURM_CPU_FREQ_REQ to match sbatch. > -- Add missing bash-completions dependency to slurm-smd-client debian package. > -- Fix bash-completions installation in debian pacakges. > -- Fix GRES environment variable indices being incorrect when only using a > subset of all GPUs on a node and the --gres-flags=allow-task-sharing option > -- Add missing mariadb/mysql client package dependency to debian package. > -- Fail the debian package build early if mysql cannot be found. > -- Prevent scontrol from segfaulting when requesting scontrol show reservation > --json or --yaml if there is an error retrieving reservations from the > slurmctld. > -- switch/hpe_slingshot - Fix security issue around managing VNI access. > -- switch/nvidia_imex - Fix security issue managing IMEX channel access. > -- switch/nvidia_imex - Allow for compatibility with job_container/tmpfs. > * Changes in Slurm 23.11.9 > ========================== > -- Fix many commands possibly reporting an "Unexpected Message Received" when > in reality the connection timed out. > -- Fix heterogeneous job components not being signaled with scancel --ctld and > 'DELETE slurm/v0.0.40/jobs' if the job ids are not explicitly given, > the heterogeneous job components match the given filters, and the > heterogeneous job leader does not match the given filters. > -- Fix regression from 23.02 impeding job licenses from being cleared. > -- Move error to log_flag which made _get_joules_task error to be logged to the > user when too many rpcs were queued in slurmd for gathering energy. > -- slurmrestd - Prevent a slurmrestd segfault when modifying an association > without specifying max TRES limits in the request if those TRES > limits are currently defined in the association. This affects the following > fields of endpoint 'POST /slurmdb/v0.0.38/associations/': > 'associations/max/tres/per/job' > 'associations/max/tres/per/node' > 'associations/max/tres/total' > 'associations/max/tres/minutes/per/job' > 'associations/max/tres/minutes/total' > -- Fix power_save operation after recovering from a failed reconfigure. > -- scrun - Delay shutdown until after start requested. This caused scrun > to never start or shutdown and hung forever when using --tty. > -- Fix backup slurmctld potentially not running the agent when taking over as > the primary controller. > -- Fix primary controller not running the agent when a reconfigure of the > slurmctld fails. > -- jobcomp/{elasticsearch,kafka} - Avoid sending fields with invalid date/time. > -- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when > more than 10 threads try to get energy at the same time. This prevented > the possibility to get energy from slurmd by any step until slurmd was > restarted, so losing energy accounting metrics in the node. > -- slurmrestd - Fix memory leak for dbv0.0.39 jobs query which occurred if the > query parameters specified account, association, cluster, constraints, > format, groups, job_name, partition, qos, reason, reservation, state, users, > or wckey. This affects the following endpoints: > 'GET /slurmdb/v0.0.39/jobs' > -- switch/hpe_slingshot - Fix security issue around managing VNI access. > * Changes in Slurm 23.02.8 > ========================== > -- Fix rare deadlock when a dynamic node registers at the same time that a > once per minute background task occurs. > -- Fix assertion in developer mode on a failed message unpack. > -- switch/hpe_slingshot - Fix security issue around managing VNI access.

1 0

slurmd error: port already in use, resulting in slaves not being able to communicate with master slurmctld
by Shooktija S N 30 Jul '24

30 Jul '24

Hi, I'm trying to set up a Slurm (version 22.05.8) cluster consisting of 3 nodes with these hostnames and local IP addresses: server1 - 10.36.17.152 server2 - 10.36.17.166 server3 - 10.36.17.132 I had scrambled together a minimum working example using these resources: https://github.com/SergioMEV/slurm-for-dummies https://blog.devops.dev/slurm-complete-guide-a-to-z-concepts-setup-and-trou… For a while everything looked fine and I was able to run the command I usually use to see if everything is fine: srun --label --nodes=3 hostname Which used to show the expected output of the hostnames of all 3 computers, namely: server1, server2, and server3. However - after having made no changes to the configs - the command no longer works if I specify the number of nodes as anything more than 1, this behaviour is consistent on all 3 computers, the output of 'sinfo' is also included below: root@server1:~# srun --nodes=1 hostnameserver1root@server1:~# root@server1:~# srun --nodes=3 hostnamesrun: Required node not available (down, drained or reserved)srun: job 312 queued and waiting for resources^Csrun: Job allocation 312 has been revokedsrun: Force Terminated JobId=312root@server1:~# root@server1:~# ssh server2 "srun --nodes=1 hostname"server1root@server1:~# root@server1:~# ssh server2 "srun --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 314 queued and waiting for resources^Croot@server1:~# root@server1:~# root@server1:~# sinfoPARTITION AVAIL TIMELIMIT NODES STATE NODELISTmainPartition* up infinite 2 down* server[2-3]mainPartition* up infinite 1 idle server1root@server1:~# Turns out, slurmctld on the master node (hostname: server1) and slurmd on the slave nodes (hostnames: server2 & server3) are throwing some errors probably related to networking: A few lines before and after the first occurence of the error in slurmctld.log on the master node - it's the only type of error I have noticed in the logs (pastebin to the entire log <https://pastebin.com/GBSWXZJR>): root@server1:/var/log# grep -B 20 -A 5 -m1 -i "error" slurmctld.log[2024-07-26T13:13:49.579] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions[2024-07-26T13:13:49.580] debug: power_save module disabled, SuspendTime < 0[2024-07-26T13:13:49.580] Running as primary controller[2024-07-26T13:13:49.580] debug: No backup controllers, not launching heartbeat.[2024-07-26T13:13:49.580] debug: priority/basic: init: Priority BASIC plugin loaded[2024-07-26T13:13:49.580] No parameter for mcs plugin, default values set[2024-07-26T13:13:49.580] mcs: MCSParameters = (null). ondemand set.[2024-07-26T13:13:49.580] debug: mcs/none: init: mcs none plugin loaded[2024-07-26T13:13:49.580] debug2: slurmctld listening on 0.0.0.0:6817[2024-07-26T13:13:52.662] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded[2024-07-26T13:13:52.662] debug2: Processing RPC: MESSAGE_NODE_REGISTRATION_STATUS from UID=0[2024-07-26T13:13:52.662] debug: gres/gpu: init: loaded[2024-07-26T13:13:52.662] debug: validate_node_specs: node server1 registered with 0 jobs[2024-07-26T13:13:52.662] debug2: _slurm_rpc_node_registration complete for server1 usec=229[2024-07-26T13:13:53.586] debug: Spawning registration agent for server[2-3] 2 hosts[2024-07-26T13:13:53.586] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2[2024-07-26T13:13:53.586] debug: sched: Running job scheduler for default depth.[2024-07-26T13:13:53.586] debug2: Spawning RPC agent for msg_type REQUEST_NODE_REGISTRATION_STATUS[2024-07-26T13:13:53.587] debug2: Tree head got back 0 looking for 2[2024-07-26T13:13:53.588] debug2: _slurm_connect: failed to connect to 10.36.17.166:6818: Connection refused[2024-07-26T13:13:53.588] debug2: Error connecting slurm stream socket at 10.36.17.166:6818: Connection refused[2024-07-26T13:13:53.588] debug2: _slurm_connect: failed to connect to 10.36.17.132:6818: Connection refused[2024-07-26T13:13:53.588] debug2: Error connecting slurm stream socket at 10.36.17.132:6818: Connection refused[2024-07-26T13:13:54.588] debug2: _slurm_connect: failed to connect to 10.36.17.166:6818: Connection refused[2024-07-26T13:13:54.588] debug2: Error connecting slurm stream socket at 10.36.17.166:6818: Connection refused[2024-07-26T13:13:54.589] debug2: _slurm_connect: failed to connect to 10.36.17.132:6818: Connection refused The connections to 10.36.17.166:6818 and 10.36.17.132:6818 are refused. Those are ports specified by the 'SlurmdPort' key in slurm.conf There are similar errors in the slurmd.log files on both the slave nodes as well: slurmd.log on server2, the error is only at the end of the file (pastebin to the entire log <https://pastebin.com/TwSMiAp7>): root@server2:/var/log# tail -5 slurmd.log [2024-07-26T13:13:53.018] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.018] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.018] debug2: No mpi.conf file (/etc/slurm/mpi.conf)[2024-07-26T13:13:53.018] error: Error binding slurm stream socket: Address already in use[2024-07-26T13:13:53.018] error: Unable to bind listen port (6818): Address already in use slurmd.log on server3 (pastebin to the entire log <https://pastebin.com/K55cAGLb>): root@server3:/var/log# tail -5 slurmd.log [2024-07-26T13:13:53.383] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.383] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.383] debug2: No mpi.conf file (/etc/slurm/mpi.conf)[2024-07-26T13:13:53.384] error: Error binding slurm stream socket: Address already in use[2024-07-26T13:13:53.384] error: Unable to bind listen port (6818): Address already in use I use this script to restart slurm whenever I change any of the configs, could the order in which these operations are being done cause the problems I'm facing: #! /bin/bashscp /etc/slurm/slurm.conf /etc/slurm/gres.conf server2:/etc/slurm/ && echo copied slurm.conf and gres.conf to server2;scp /etc/slurm/slurm.conf /etc/slurm/gres.conf server3:/etc/slurm/ && echo copied slurm.conf and gres.conf to server3;echoecho restarting slurmctld and slurmd on server1(scontrol shutdown ; sleep 3 ; rm -f /var/log/slurmd.log /var/log/slurmctld.log ; slurmctld -d ; sleep 3 ; slurmd) && echo doneecho restarting slurmd on server2(ssh server2 "rm -f /var/log/slurmd.log /var/log/slurmctld.log ; slurmd") && echo doneecho restarting slurmd on server3(ssh server3 "rm -f /var/log/slurmd.log /var/log/slurmctld.log ; slurmd") && echo done Config files: slurm.conf without the comments: root@server1:/etc/slurm# grep -v "#" slurm.conf ClusterName=DlabClusterSlurmctldHost=server1GresTypes=gpuProctrackType=proctrack/linuxprocReturnToService=1SlurmctldPidFile=/var/run/slurmctld.pidSlurmctldPort=6817SlurmdPidFile=/var/run/slurmd.pidSlurmdPort=6818SlurmdSpoolDir=/var/spool/slurmdSlurmUser=rootStateSaveLocation=/var/spool/slurmctldTaskPlugin=task/affinity,task/cgroupInactiveLimit=0KillWait=30MinJobAge=300SlurmctldTimeout=120SlurmdTimeout=300Waittime=0SchedulerType=sched/backfillSelectType=select/cons_tresJobCompType=jobcomp/noneJobAcctGatherFrequency=30SlurmctldDebug=debug2SlurmctldLogFile=/var/log/slurmctld.logSlurmdDebug=debug2SlurmdLogFile=/var/log/slurmd.logNodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP gres.conf: root@server1:/etc/slurm# cat gres.confNodeName=server1 Name=gpu File=/dev/nvidia0NodeName=server2 Name=gpu File=/dev/nvidia0NodeName=server3 Name=gpu File=/dev/nvidia0 These config files are the same on all 3 computers. As a complete beginner to Linux and Slurm administration, I have been struggling to understand even the most basic documentation, and I have been unable to find answers online. Any assistance would be greatly appreciated. Thanks!

1 1

slurmctld hourly: Unexpected missing socket error
by Jason Ellul 30 Jul '24

30 Jul '24

Hi all, I am hoping someone can help with our problem. Every hour after restarting slurmctld the controller becomes unresponsive to commands for 1 sec, reporting errors such as: [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934767]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934760]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934875]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934906]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[939016]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error It occurs consistently at around the hour mark, but generally not at other times, unless we run a reconfigure or restart the controller. We don’t see any issues in the slurmdbd.log and the errors are also always msg type RESPONSE. We have tried building a new server on different infrastructure, but the problem has persisted. Yesterday we even tried updating slurm to v24.05.1 in the hope that may provide a fix. During our troubleshooting we have: Set: * SchedulerParameters = max_rpc_cnt=400,sched_min_interval=50000,sched_max_job_start=300,batch_sched_delay=20,bf_resolution=600,bf_min_prio_reserve=2000,bf_min_age_reserve=600 * SlurmctldPort = 6808-6817 But although the stats in sdiag have improved we still see the errors. On our monitoring software we also see a drop in network and disk activity during this 1 second, always at approx. 1 hour after restarting the controller. Many Thanks in advance Jason Jason Ellul Head - Research Computing Facility Office of Cancer Research Peter MacCallum Cancer Centre

2 4

Final Call for SLUG Standard Registration
by Victoria Hobson 29 Jul '24

29 Jul '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information, abstracts, and travel recommendations can be found here:https://slug24.splashthat.com/ The last day to register with standard pricing ($900) is this Friday, August 2nd. After this, final registration will run until August 30th at a price of $1100. SLUG is the best way to interact with the Slurm community and to interact with the SchedMD Support & Training staff. Don't forget to register. We can't wait to see you in Oslo! -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

Convergence of Kube and Slurm?
by Dan Healy 29 Jul '24

29 Jul '24

Bright Cluster Manager has some verbiage on their marketing site that they can manage a cluster running both Kubernetes and Slurm. Maybe I misunderstood it. But nevertheless, I am encountering groups more frequently that want to run a stack of containers that need private container networking. What’s the current state of using the same HPC cluster for both Slurm and Kube? Note: I’m aware that I can run Kube on a single node, but we need more resources. So ultimately we need a way to have Slurm and Kube exist in the same cluster, both sharing the full amount of resources and both being fully aware of resource usage. Thanks, Daniel Healy

5 4

Slurm fails before nvidia-smi command
by Aziz Ogutlu 29 Jul '24

29 Jul '24

Hi there all, We have Dell server with 2 x Nvidia H100 and running slurm on it. After restart server if we do not write nvidia-smi command slurm fails. When we run nvidia-smi && systemctl restart slurmd && systemctl restart slurmctld , slurm queue begins. Do you have any idea about this error and what can we do for this issue? -- Best regards, Aziz Öğütlü Eduline Bilişim Sanayi ve Ticaret Ltd. Şti. www.eduline.com.tr Merkez Mah. Ayazma Cad. No:37 Papirus Plaza Kat:6 Ofis No:118 Kağıthane - İstanbul - Türkiye 34406 Tel : +90 212 324 60 61 Cep: +90 541 350 40 72

4 4

Slurm sacct ResvCPURAW invalid field in version 24.12.5
by Mike Mikailov 29 Jul '24

29 Jul '24

Dear All, Does anyone know what is the equivalent field for ResvCPURAW of sacct command of Slurm in its version 23.11.5. I get: sacct: error: Invalid field requested “ResvCPURAW”. Your help at your earliest is very much appreciated. Best, -Mike Sent from my iPhone

2 1

Background tasks in Slurm scripts?
by Steffen Grunewald 26 Jul '24

26 Jul '24

Good morning, yesterday I came across a Slurm (sbatch) script that, after doing some stuff in the foreground, runs another executable in the background - and doesn't "wait" for it to finish - literally the last line of the script is executable & (and that executable is supposed to take several 10 seconds or more to finish) How would Slurm handle this? Will the end of the script immediately trigger the job epilog, and what would happen to the leftover task? This certainly is discussed somewhere in the manual pages and other documentation but up to now I failed to find that place... Thanks, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

3 3

API - Specify GPUs
by jpuerto＠psc.edu 26 Jul '24

26 Jul '24

Good Afternoon All, It does not seem that the REST API allows for folks to configure their jobs to utilize GPUs, using the traditional methods. IE, there does not appear to be an equivalent between the --gpus (or --gres) flag on sbatch/srun and the REST API's job submission endpoint. Can anyone point me towards what should be used if the version we are on does not support tres specifications? Best regards, Juan Puerto

2 1

Error binding slurm stream socket: Address already in use, and GPU GRES verification
by Shooktija S N 23 Jul '24

23 Jul '24

Hi, I am trying to set up Slurm with GPUs as GRES on a 3 node configuration (hostnames: server1, server2, server3). For a while everything looked fine and I was able to run srun --label --nodes=3 hostname which is what I use to test if Slurm is working correctly, and then it randomly stops. Turns out slurmctld is not working and it throws the following error (the two lines are consecutive in the log file): root@server1:/var/log# grep -i error slurmctld.log [2024-07-22T14:47:32.302] error: Error binding slurm stream socket: Address already in use[2024-07-22T14:47:32.302] fatal: slurm_init_msg_engine_port error Address already in use This error is being thrown after having made no changes to the config files, in fact the cluster wasn't used at all for a few weeks before this error was thrown. This is the simple script I use to restart Slurm: root@server1:~# cat slurmRestart.sh #! /bin/bashscp /etc/slurm/slurm.conf server2:/etc/slurm/ && echo copied slurm.conf to server2;scp /etc/slurm/slurm.conf server3:/etc/slurm/ && echo copied slurm.conf to server3;rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld ; echo restarting slurm on server1;(ssh server2 "rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld") && echo restarting slurm on server2;(ssh server3 "rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld") && echo restarting slurm on server3; Could the error be due to the slurmd and/or slurmctld not being started in the right order? The other question I have is regarding the configuration of a GPU as a GRES - how do I verify that it has been configured correctly? I was told to use srun nvidia-smi with and without having enabled GPU use, but whether or not I enable GPU usage has no effect on the output of the command: root@server1:~# srun --nodes=1 nvidia-smi --query-gpu=uuid --format=csvuuidGPU-55f127a8-dbf4-fd12-3cad-c0d5f2dcb005root@server1:~# root@server1:~# srun --nodes=1 --gpus-per-node=1 nvidia-smi --query-gpu=uuid --format=csvuuidGPU-55f127a8-dbf4-fd12-3cad-c0d5f2dcb005 I am sceptical if about whether the GPU has properly been configured, is this the best way to check if it has? *The error:* I first noticed this happening when I tried to run the command I usually use to see if everything is fine, the srun command runs only one node, and the only way to stop it if I specify the number of nodes as 3 is to press Ctrl+C: root@server1:~# srun --label --nodes=1 hostname0: server1root@server1:~# ssh server2 "srun --label --nodes=1 hostname"0: server1root@server1:~# ssh server3 "srun --label --nodes=1 hostname"0: server1root@server1:~# srun --label --nodes=3 hostnamesrun: Required node not available (down, drained or reserved)srun: job 265 queued and waiting for resources^Csrun: Job allocation 265 has been revokedsrun: Force Terminated JobId=265root@server1:~# ssh server2 "srun --label --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 266 queued and waiting for resources^Croot@server1:~# ssh server3 "srun --label --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 267 queued and waiting for resourcesroot@server1:~# *The logs:* 1) The last 30 lines of */var/log/slurmctld.log* at the debug5 level in server #1 (pastebin to the entire log <https://pastebin.com/fw4C4xtr>): root@server1:/var/log# tail -30 slurmctld.log [2024-07-22T14:47:32.301] debug: Updating partition uid access list[2024-07-22T14:47:32.301] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/resv_state` as buf_t[2024-07-22T14:47:32.301] debug3: Version string in resv_state header is PROTOCOL_VERSION[2024-07-22T14:47:32.301] Recovered state of 0 reservations[2024-07-22T14:47:32.301] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/trigger_state` as buf_t[2024-07-22T14:47:32.301] State of 0 triggers recovered[2024-07-22T14:47:32.301] read_slurm_conf: backup_controller not specified[2024-07-22T14:47:32.301] select/cons_tres: select_p_reconfigure: select/cons_tres: reconfigure[2024-07-22T14:47:32.301] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions[2024-07-22T14:47:32.301] debug: power_save module disabled, SuspendTime < 0[2024-07-22T14:47:32.301] Running as primary controller[2024-07-22T14:47:32.301] debug: No backup controllers, not launching heartbeat.[2024-07-22T14:47:32.301] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/priority_basic.so[2024-07-22T14:47:32.301] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Priority BASIC plugin type:priority/basic version:0x160508[2024-07-22T14:47:32.301] debug: priority/basic: init: Priority BASIC plugin loaded[2024-07-22T14:47:32.301] debug3: Success.[2024-07-22T14:47:32.301] No parameter for mcs plugin, default values set[2024-07-22T14:47:32.301] mcs: MCSParameters = (null). ondemand set.[2024-07-22T14:47:32.301] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/mcs_none.so[2024-07-22T14:47:32.301] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:mcs none plugin type:mcs/none version:0x160508[2024-07-22T14:47:32.301] debug: mcs/none: init: mcs none plugin loaded[2024-07-22T14:47:32.301] debug3: Success.[2024-07-22T14:47:32.302] debug3: _slurmctld_rpc_mgr pid = 3159324[2024-07-22T14:47:32.302] debug3: _slurmctld_background pid = 3159324[2024-07-22T14:47:32.302] error: Error binding slurm stream socket: Address already in use[2024-07-22T14:47:32.302] fatal: slurm_init_msg_engine_port error Address already in use[2024-07-22T14:47:32.304] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.304] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.304] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.304] slurmscriptd: debug: _slurmscriptd_mainloop: finished 2) Entirety of *slurmctld.log on server #2*: root@server2:/var/log# cat slurmctld.log [2024-07-22T14:47:32.614] debug: slurmctld log levels: stderr=debug5 logfile=debug5 syslog=quiet[2024-07-22T14:47:32.614] debug: Log file re-opened[2024-07-22T14:47:32.615] slurmscriptd: debug: slurmscriptd: Got ack from slurmctld, initialization successful[2024-07-22T14:47:32.615] slurmscriptd: debug: _slurmscriptd_mainloop: started[2024-07-22T14:47:32.616] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.616] debug: slurmctld: slurmscriptd fork()'d and initialized.[2024-07-22T14:47:32.616] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.616] debug: _slurmctld_listener_thread: started listening to slurmscriptd[2024-07-22T14:47:32.616] debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.616] debug3: Called _msg_readable[2024-07-22T14:47:32.616] slurmctld version 22.05.8 started on cluster dlabcluster[2024-07-22T14:47:32.616] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/cred_munge.so[2024-07-22T14:47:32.616] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x160508[2024-07-22T14:47:32.616] cred/munge: init: Munge credential signature plugin loaded[2024-07-22T14:47:32.616] debug3: Success.[2024-07-22T14:47:32.616] error: This host (server2/server2) not a valid controller[2024-07-22T14:47:32.617] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.617] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.617] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.617] slurmscriptd: debug: _slurmscriptd_mainloop: finished 3) Entirety of *slurmctld.log on server #3*: root@server3:/var/log# cat slurmctld.log [2024-07-22T14:47:32.927] debug: slurmctld log levels: stderr=debug5 logfile=debug5 syslog=quiet[2024-07-22T14:47:32.927] debug: Log file re-opened[2024-07-22T14:47:32.928] slurmscriptd: debug: slurmscriptd: Got ack from slurmctld, initialization successful[2024-07-22T14:47:32.928] slurmscriptd: debug: _slurmscriptd_mainloop: started[2024-07-22T14:47:32.928] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.928] debug: slurmctld: slurmscriptd fork()'d and initialized.[2024-07-22T14:47:32.928] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.928] slurmctld version 22.05.8 started on cluster dlabcluster[2024-07-22T14:47:32.929] debug: _slurmctld_listener_thread: started listening to slurmscriptd[2024-07-22T14:47:32.929] debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.929] debug3: Called _msg_readable[2024-07-22T14:47:32.929] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/cred_munge.so[2024-07-22T14:47:32.929] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x160508[2024-07-22T14:47:32.929] cred/munge: init: Munge credential signature plugin loaded[2024-07-22T14:47:32.929] debug3: Success.[2024-07-22T14:47:32.929] error: This host (server3/server3) not a valid controller[2024-07-22T14:47:32.930] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.930] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.930] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.930] slurmscriptd: debug: _slurmscriptd_mainloop: finished *The config files (shared by all 3 computers):* 1) */etc/slurm/slurm.conf* without the comments: root@server1:/etc/slurm# grep -v "#" slurm.conf ClusterName=DlabClusterSlurmctldHost=server1GresTypes=gpuProctrackType=proctrack/linuxprocReturnToService=1SlurmctldPidFile=/var/run/slurmctld.pidSlurmctldPort=6817SlurmdPidFile=/var/run/slurmd.pidSlurmdPort=6818SlurmdSpoolDir=/var/spool/slurmdSlurmUser=rootStateSaveLocation=/var/spool/slurmctldTaskPlugin=task/affinity,task/cgroupInactiveLimit=0KillWait=30MinJobAge=300SlurmctldTimeout=120SlurmdTimeout=300Waittime=0SchedulerType=sched/backfillSelectType=select/cons_tresJobCompType=jobcomp/noneJobAcctGatherFrequency=30SlurmctldDebug=debug5SlurmctldLogFile=/var/log/slurmctld.logSlurmdDebug=debug5SlurmdLogFile=/var/log/slurmd.logNodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP 2) */etc/slurm/gres.conf*: root@server1:/etc/slurm# cat gres.conf NodeName=server1 Name=gpu File=/dev/nvidia0NodeName=server2 Name=gpu File=/dev/nvidia0NodeName=server3 Name=gpu File=/dev/nvidia0 These files are the same on all 3 computers: root@server1:/etc/slurm# diff slurm.conf <(ssh server2 "cat /etc/slurm/slurm.conf")root@server1:/etc/slurm# diff slurm.conf <(ssh server3 "cat /etc/slurm/slurm.conf")root@server1:/etc/slurm# diff gres.conf <(ssh server2 "cat /etc/slurm/gres.conf")root@server1:/etc/slurm# diff gres.conf <(ssh server3 "cat /etc/slurm/gres.conf")root@server1:/etc/slurm# Thank you, Shookti

1 0

Cgroup
by stth 22 Jul '24

22 Jul '24

Hello, I am configuring cgroups on my server for the first time. I've created a cgroup.conf file in the Slurm directory with the following values: ConstrainCores=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes AllowedRAMSpace=90 AllowedSwapSpace=10 I feel like this configuration might be incomplete. Can anyone provide assistance? I haven't found any specific guidance online. Thanks!

2 1

Custom Plugin Integration
by Bhaskar Chakraborty 20 Jul '24

20 Jul '24

Hello, We wish to have a schedulingintegration with Slurm. Our own application has a backend system which willdecide the placement of jobs across hosts & CPU cores. The backend takesits own time to come back with a placement (which may take a few seconds) & we expect slurm to update it regularly about any change in the current state ofavailable resources. For this we believe we have 3 options broadly: - We use the const_tres Select plugin & modify it to let it query our backend system for job placements. - We write our own Select plugin avoiding any other Select plugin. - We use existing select plugin & also register our own plugin. Idea is that our plugin will cater to 'our' jobs (specific partition, say) while all other jobs would be taken up by the default plugin. Problem with a> is that this leads to modification ofexisting plugin code & calling (our) library code from inside Select pluginlib. With b> the issue is unless we have the full Slurm cluster toourselves this isn't viable. Any insight how to proceed with this? Where shouldour select plugin, assuming we need to make one, fits in the slurm integration. We are not sure whether c> is allowed in Slurm. We went through existing Select plugins Linear & cons_tres.However, not able to figure out how to use them or write something on similarlines to suit our purpose. Any help in this regard is appreciated. Apologies if this question (or any other very similar) is already answered,please point to the relevant thread then. Thanks in advance for any pointers. Regards, Bhaskar.

3 7

CLOUD nodes with unknown IP addresses
by Martin Lee 19 Jul '24

19 Jul '24

I've read the following in the slurm power saving docs: https://slurm.schedmd.com/power_save.html > *cloud_dns* > By default, Slurm expects that the network addresses for cloud nodes won't be known until creation of the node and that Slurm will be notified of the node's address upon registration. Since Slurm communications rely on the node configuration found in the slurm.conf, Slurm will tell the client command, after waiting for all nodes to boot, each node's IP address. However, in environments where the nodes are in DNS, this step can be avoided by configuring this option. > I am creating the nodes on demand and don't know the IP ahead of the instance start, so cloud_dns is not set. I'm confused specifically by "Slurm will be notified of the node's address upon registration." Who/what is expected to do this? If it is expected to be performed by the ResumeProgram, does it need to be done before slurmd starts on the node? Is it OK if the node does it after slurmd has started with something like: scontrol update nodename=$(hostname -s) nodeaddr=$(hostname -I) nodehostname=$(hostname) scontrol reconfigure Thank you, Martin

2 2

Slurm Users cannot open gui login nodes with X11 without logging into the Compute node.
by Alper AYKUT 18 Jul '24

18 Jul '24

Hello 1 Login Server I have a total of 3 calculation nodes. Especially in non-GUI jobs, I can run scripts successfully with Slurm sbatch. However, when it is with Gui, I run it with X11. However, my problem for now is this: When I run sbatch gui.sh, it runs smoothly and automatically selects the calculation node and gui is active on the login server. But the node it selects must be logged in with the same user for x11 to work. How can I run the application with gui on the login node without logging in on the calculation node? The calculation node gives the following error when I run the script without logging in. How can I solve this problem. srun: error: Ignoring --x11 option for a job step within an existing job. Set x11 options at job allocation time. srun: error: tutelservernode01: task 0: Exited with exit code 255 My test Script ; #!/bin/bash #SBATCH --partition=debug #Kullanılacak partition (compute, debug, vb.) #SBATCH --job-name=xclock # İşin adı #SBATCH --output=xclock-%j.log # İşin çıktı dosyası #SBATCH --error=xclock-%j.err # İşin hata çıktı dosyası # Slurm'un hangi node'dan işlemi çalıştıracağını seçmesine izin verin # --nodelist parametresi kullanılmazsa, Slurm otomatik olarak bir node seçecektir. # xclock komutunu başlatmak için DISPLAY ortam değişkenini ayarlayın #export DISPLAY=localhost:10.0 # MobaXterm veya başka bir X sunucusunun adresine göre ayarlayın export DISPLAY=$DISPLAY # xclock komutunu sbatch içinde başlatın srun --x11 xclock thankyou for support.

1 0

Nodes TRES double what is requested
by jack.mellor＠mbda.co.uk 12 Jul '24

12 Jul '24

Hi, We are running slurm 23.02.6. Our nodes have hyperthreading disabled and we have slurm.conf set to CPU=32 for each node (each node has 2 processes with 16 cores). When we allocated a job, such as salloc -n 32, it will allocate a whole node but using sinfo shows double the allocation in the TRES=64. It also shows in sinfo that the node has 4294967264 idle CPUs. Not sure if its a known bug, or an issue with our config? I have tried various things, like setting the sockets/boards in slurm.conf. Thanks Jack

5 4

SLURM noob administrator question
by Cutts, Tim 12 Jul '24

12 Jul '24

Still learning about SLURM, so please forgive me if I ask a naïve question I like to use Anders Halager’s gnodes command to visualise the state of our nodes. I’ve noticed lately that we fairly often see things like this (apologies for line wrap): +- core - 46 cores & 186GB -----------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+ | seskscpn301 0G ___OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO | seskscpn309 172G ..........................................!!!! | seskscpn317 0G OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO | Now, you can see in this that nodes 301 and 317 are more or less fully loaded. This is great. But 309 is in an interesting state. Four overloaded cores, and all other cores unused, and plenty of RAM available. And yet SLURM is not scheduling any more work to that node. Right now there are more than 2000 jobs pending, many of which could run on that node. But SLURM is not scheduling them, and I don’t know why. One thing I’ve seen cause this is a job trying to use more CPUs than it has been allocated. The cgroup stops this being a real problem of course, but it does cause the load average to go high. Is this what’s causing SLURM to stop sending anything to the node? Is there a configuration change that might help in this situation? Thanks in advance, Tim -- Tim Cutts Scientific Computing Platform Lead AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>

2 1

Replacing MUNGE with SACK (auth/slurm)
by Daniel Letai 11 Jul '24

11 Jul '24

Does SACK replace MUNGE? As in - MUNGE is not required when building Slurm or on compute? If so, can the Requires and BuildRequires for munge be made optional on bcond_without_munge in the spec file? Or is there a reason MUNGE must remain a hard require for Slurm? Thanks, --Dani_L.

1 0

Job submitted to multiple partitions not running when any partition is full
by Paul Raines 09 Jul '24

09 Jul '24

I have a job 465072 submitted to multiple partitions (rtx6000,rtx8000,pubgpu) JOBID PARTITION PENDING PRIORITY TRES_ALLOC|REASON 4650727 rtx6000 47970 0.00367972 cpu=5,mem=400G,node=1,gpu=1|Priority 4650727 rtx8000 47970 0.00367972 cpu=5,mem=400G,node=1,gpu=1|Priority 4650727 pubgpu 47970 0.00367972 cpu=5,mem=400G,node=1,gpu=1|Priority 4646926 rtx6000 487048 0.00121987 cpu=10,mem=32G,node=1,gpu=1|Priority,Resources 4650186 rtx8000 56979 0.00000000 cpu=4,mem=10G,node=1,gpu=1|Priority,Resources We see the two partitions rtx6000 and rtx8000 are full and two other jobs are at the top of the queue waiting to run on those. But partition pubgpu is NOT full and you can see here a node leo with resources to run the 4650727 job HOST PARTITION CORES MEMORY GPUS leo pubgpu 48/ 64 12288/1030994 0/ 1 leo pubcpu 48/ 64 12288/1030994 0/ 1 The node leo is NOT part of the rtx6000 or rtx8000 partitions and there are no other pending jobs waiting on either the pubgpu or pubcpu partition that leo is part of So why is 4650727 not running on the pubgpu partition? --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

2 2

Temporarily bypassing pam_slurm_adopt.so
by Daniel L'Hommedieu 09 Jul '24

09 Jul '24

Hi, all. We have a use case where we need to allow a group of users (members of an LDAP group, which I can easily add to a Linux group) to SSH to a compute node, without disabling pam_slurm_adopt.so. Is there a way to do this? We can add users to the sudo group, which will bypass pam_slurm_adopt.so, but we do not want to grant sudo access to these users. Is there any way to bypass for a group of users, pam_slurm_adopt.so without removing it from /etc/pam.d/sshd, and without adding them to the sudo group? Thanks. Daniel

5 4

vers 23: slurmctld pb (memory leak and response time)
by LEROY Christine 208562 09 Jul '24

09 Jul '24

Hello Everyone, We tried several version of slurm 23-x (till the last one: 23.11.8), and we observe a memory leak in slurmctld in addition to a bad response time (sometimes). It seems to be due to a slurm.conf configuration with Partition containing Allowgroups option. Is it a known problem? Is it recommended to use the version 22, or 24 ? Thanks in advance, Christine Leroy

1 1

Can SLURM queue different jobs to start concurrently?
by Dan Healy 08 Jul '24

08 Jul '24

Hi there, I've received a question from an end user, which I presume the answer is "No", but would like to ask the community first. Scenario: The user wants to create a series of jobs that all need to start at the same time. Example: there are 10 different executable applications which have varying CPU and RAM constraints, all of which need to communicate via TCP/IP. Of course the user could design some type of idle/statusing mechanism to wait until all jobs are *randomly *started, then begin execution, but this feels like a waste of resources. The complete execution of these 10 applications would be considered a single simulation. The goal would be to distribute these 10 applications across the cluster and not necessarily require them all to execute on a single node. Is there a good architecture for this using SLURM? If so, please kindly point me in the right direction. -- Thanks, Daniel Healy

3 2

Re: Can SLURM queue different jobs to start concurrently?
by Lloyd Brown 08 Jul '24

08 Jul '24

I'm confused. Why can't they just use a multi-node job, and have the job script farm out the individual tasks to the various workers through some mechanism (srun, mpirun, ssh, etc.)? AFAIK, there's nothing preventing a job from using resources on multiple hosts. The job just needs to have some way of pushing the work out to those hosts. Lloyd On 7/8/24 14:17, Dan Healy via slurm-users wrote: > Hi there, > > I've received a question from an end user, which I presume the answer > is "No", but would like to ask the community first. > > Scenario: The user wants to create a series of jobs that all need to > start at the same time. Example: there are 10 different executable > applications which have varying CPU and RAM constraints, all of which > need to communicate via TCP/IP. Of course the user could design some > type of idle/statusing mechanism to wait until all jobs are /randomly > /started, then begin execution, but this feels like a waste of > resources. The complete execution of these 10 applications would be > considered a single simulation. The goal would be to distribute these > 10 applications across the cluster and not necessarily require them > all to execute on a single node. > > Is there a good architecture for this using SLURM? If so, please > kindly point me in the right direction. > > -- > Thanks, > > Daniel Healy -- Lloyd Brown HPC Systems Administrator Office of Research Computing Brigham Young University http://rc.byu.edu

1 0

cgroups/v2 plugin rpmbuild issue
by Chris Taylor 06 Jul '24

06 Jul '24

Trying to use rpmbuild on Rocky9 Linux, Slurm 21.08 - I want to build with cgroups/v2 support and have these installed: libbpf-devel.x86_64 2:1.3.0-2.el9 dbus-devel.x86_64 1:1.12.20-8.el9 kernel-headers.x86_64 5.14.0-427.22.1.el9_4 hwloc-devel.x86_64 2.4.1-5.el9 In https://slurm.schedmd.com/cgroup_v2.html it says: Requirements For building cgroup/v2 there are two required libraries checked at configure time... Look at your config.log when configuring to see if they were correctly detected on your system. I don't see any mention of 'ebpf', 'bpf', or 'dbus' when I grep through ~/rpmbuild/BUILD/slurm-21.08.8-2/config.log. Before I noticed that, I installed the RPMS on my cluster and get cgroup errors when I try to run a job or get an allocation. The slurmctld and slurmd processes start up fine on the servers, and I have a generic /etc/slurm/cgroup.conf. It seems like the cgroups plugin isn't getting built right with rpmbuild- how do I troubleshoot? Thanks Chris

2 3

Qlustar HPC Core Stack 24.06
by Roland Fehrenbacher 06 Jul '24

06 Jul '24

Hi all, the Qlustar HPC Core Stack 24.06 update is available. It includes an update to Slurm 23.11.8. Find further details at https://qlustar.com/news/2024/2024-06-29-hpc-core-stack-2406 Enjoy, Roland ------- https://qlustar.com -- 100% Open Source HPC / AI / Storage / Cloud Linux Cluster OS --

1 0

Run a program via Strigger when a node joins the cluster
by Karri Vrkreddy 06 Jul '24

06 Jul '24

Hi, We have a requirement to run a specific program whenever any new node joins the slurm cluster. For this, we have tried using strigger with the following options : "strigger --set --node --up --flags=perm --program=<program>" We see that the trigger is not getting activated always. Checked slurmctld logs for "type=node:up" and noticed it is not always happening. Based on documentation at https://slurm.schedmd.com/strigger.html , "up" flag would mean Trigger an event if the specified node is returned to service from a DOWN state 1. From Slurm Workload Manager - Slurm Power Saving Guide , and the above message does it mean that a node has to be already in DOWN state in the cluster for a trigger with "--up" to get activated ? 2. Are there any other ways beyond strigger to achieve this ? | | | | | | | | | | | Slurm Workload Manager - Slurm Power Saving Guide | | | -- Thanks Karri

1 0

Using sharding
by Ricardo Cruz 05 Jul '24

05 Jul '24

Greetings, There are not many questions regarding GPU sharding here, and I am unsure if I am using it correctly... I have configured it according to the instructions <https://slurm.schedmd.com/gres.html>, and it seems to be configured properly: $ scontrol show node compute01 NodeName=compute01 Arch=x86_64 CoresPerSocket=32 CPUAlloc=48 CPUEfctv=128 CPUTot=128 CPULoad=10.95 AvailableFeatures=(null) ActiveFeatures=(null) * Gres=gpu:8,shard:32* [truncated] When running with gres:gpu everything works perfectly: $ /usr/bin/srun --gres=gpu:2 ls srun: job 192 queued and waiting for resources srun: job 192 has been allocated resources (...) However, when using sharding, it just stays waiting indefinitely: $ /usr/bin/srun --gres=shard:2 ls srun: job 193 queued and waiting for resources The reason it gives for pending is just "Resources": $ scontrol show job 193 JobId=193 JobName=ls UserId=rpcruz(1000) GroupId=rpcruz(1000) MCS_label=N/A Priority=1 Nice=0 Account=account QOS=normal * JobState=PENDING Reason=Resources Dependency=(null)* Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2024-06-28T05:36:51 EligibleTime=2024-06-28T05:36:51 AccrueTime=2024-06-28T05:36:51 StartTime=2024-06-29T18:13:22 EndTime=2024-07-01T18:13:22 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-06-28T05:37:20 Scheduler=Backfill:* Partition=partition AllocNode:Sid=localhost:47757 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=1031887M,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=ls WorkDir=/home/rpcruz Power= * TresPerNode=gres/shard:2* Again, I think I have configured it properly - it shows up correctly in scontrol (as shown above). Our setup is pretty simple - I just added shard to /etc/slurm/slurm.conf: GresTypes=gpu,shard NodeName=compute01 Gres=gpu:8,shard:32 [truncated] Our /etc/slurm/gres.conf is also straight-forward: (it works fine for --gres=gpu:1) Name=gpu File=/dev/nvidia[0-7] Name=shard Count=32 Maybe I am just running srun improperly? Shouldn't it just be srun --gres= shard:2 to allocate half of a GPU? (since I am using 32 shards for the 8 gpus, so it's 4 shards per gpu) Thank you very much for your attention, -- Ricardo Cruz - https://rpmcruz.github.io

5 7

Re: Slurm commands fail when run in Singularity container with the error "Invalid user for SlurmUser slurm, SINGULARITYENV_SLURM_CONF
by Robert Kudyba 03 Jul '24

03 Jul '24

In https://support.schedmd.com/show_bug.cgi?id=9282#c6 Tim mentioned this env variable SINGULARITYENV_SLURM_CONF, what is the usage/syntax for it? I can't find any reference to this. I'm running into the same issue mentioned there. Thanks in advance!

2 2

Slurm version 24.05.1 is now available
by Tim Wickberg 27 Jun '24

27 Jun '24

We are pleased to announce the availability of Slurm version 24.05.1. This release addresses a number of minor-to-moderate issues since the 24.05 release was first announced a month ago. Slurm can be downloaded from https://www.schedmd.com/downloads.php . - Tim > * Changes in Slurm 24.05.1 > ========================== > -- Fix slurmctld and slurmdbd potentially stopping instead of performing a > logrotate when recieving SIGUSR2 when using auth/slurm. > -- switch/hpe_slingshot - Fix slurmctld crash when upgrading from 23.02. > -- Fix "Could not find group" errors from validate_group() when using > AllowGroups with large /etc/group files. > -- Prevent an assertion in debugging builds when triggering log rotation > in a backup slurmctld. > -- Add AccountingStoreFlags=no_stdio which allows to not record the stdio > paths of the job when set. > -- slurmrestd - Prevent a slurmrestd segfault when parsing the crontab field, > which was never usable. Now it explicitly ignores the value and emits a > warning if it is used for the following endpoints: > 'POST /slurm/v0.0.39/job/{job_id}' > 'POST /slurm/v0.0.39/job/submit' > 'POST /slurm/v0.0.40/job/{job_id}' > 'POST /slurm/v0.0.40/job/submit' > 'POST /slurm/v0.0.41/job/{job_id}' > 'POST /slurm/v0.0.41/job/submit' > 'POST /slurm/v0.0.41/job/allocate' > -- mpi/pmi2 - Fix communication issue leading to task launch failure with > "invalid kvs seq from node". > -- Fix getting user environment when using sbatch with "--get-user-env" or > "--export=" when there is a user profile script that reads /proc. > -- Prevent slurmd from crashing if acct_gather_energy/gpu is configured but > GresTypes is not configured. > -- Do not log the following errors when AcctGatherEnergyType plugins are used > but a node does not have or cannot find sensors: > "error: _get_joules_task: can't get info from slurmd" > "error: slurm_get_node_energy: Zero Bytes were transmitted or received" > However, the following error will continue to be logged: > "error: Can't get energy data. No power sensors are available. Try later" > -- sbatch, srun - Set SLURM_NETWORK environment variable if --network is set. > -- Fix cloud nodes not being able to forward to nodes that restarted with new > IP addresses. > -- Fix cwd not being set correctly when running a SPANK plugin with a > spank_user_init() hook and the new "contain_spank" option set. > -- slurmctld - Avoid deadlock during shutdown when auth/slurm is active. > -- Fix segfault in slurmctld with topology/block. > -- sacct - Fix printing of job group for job steps. > -- scrun - Log when an invalid environment variable causes the job submission > to be rejected. > -- accounting_storage/mysql - Fix problem where listing or modifying an > association when specifying a qos list could hang or take a very long time. > -- gpu/nvml - Fix gpuutil/gpumem only tracking last GPU in step. Now, > gpuutil/gpumem will record sums of all GPUS in the step. > -- Fix error in scrontab jobs when using slurm.conf:PropagatePrioProcess=1. > -- Fix slurmctld crash on a batch job submission with "--nodes 0,...". > -- Fix dynamic IP address fanout forwarding when using auth/slurm. > -- Restrict listening sockets in the mpi/pmix plugin and sattach to the > SrunPortRange. > -- slurmrestd - Limit mime types returned from query to 'GET /openapi/v3' to > only return one mime type per serializer plugin to fix issues with OpenAPI > client generators that are unable to handle multiple mime type aliases. > -- Fix many commands possibly reporting an "Unexpected Message Received" when > in reality the connection timed out. > -- Prevent slurmctld from starting if there is not a json serializer present > and the extra_constraints feature is enabled. > -- Fix heterogeneous job components not being signaled with scancel --ctld and > 'DELETE slurm/v0.0.40/jobs' if the job ids are not explicitly given, > the heterogeneous job components match the given filters, and the > heterogeneous job leader does not match the given filters. > -- Fix regression from 23.02 impeding job licenses from being cleared. > -- Move error to log_flag which made _get_joules_task error to be logged to the > user when too many rpcs were queued in slurmd for gathering energy. > -- For scancel --ctld and the associated rest api endpoints: > 'DELETE /slurm/v0.0.40/jobs' > 'DELETE /slurm/v0.0.41/jobs' > Fix canceling the final array task in a job array when the task is pending > and all array tasks have been split into separate job records. Previously > this task was not canceled. > -- Fix power_save operation after recovering from a failed reconfigure. > -- slurmctld - Skip removing the pidfile when running under systemd. In that > situation it is never created in the first place. > -- Fix issue where altering the flags on a Slurm account (UsersAreCoords) > several limits on the account's association would be set to 0 in > Slurm's internal cache. > -- Fix memory leak in the controller when relaying stepmgr step accounting to > the dbd. > -- Fix segfault when submitting stepmgr jobs within an existing allocation. > -- Added "disable_slurm_hydra_bootstrap" as a possible MpiParams parameter in > slurm.conf. Using this will disable env variable injection to allocations > for the following variables: I_MPI_HYDRA_BOOTSTRAP, > I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS, HYDRA_BOOTSTRAP, > HYDRA_LAUNCHER_EXTRA_ARGS. > -- scrun - Delay shutdown until after start requested. This caused scrun > to never start or shutdown and hung forever when using --tty. > -- Fix backup slurmctld potentially not running the agent when taking over as > the primary controller. > -- Fix primary controller not running the agent when a reconfigure of the > slurmctld fails. > -- slurmd - fix premature timeout waiting for REQUEST_LAUNCH_PROLOG with large > array jobs causing node to drain. > -- jobcomp/{elasticsearch,kafka} - Avoid sending fields with invalid date/time. > -- jobcomp/elasticsearch - Fix slurmctld memory leak from curl usage > -- acct_gather_profile/influxdb - Fix slurmstepd memory leak from curl usage > -- Fix 24.05.0 regression not deleting job hash dirs after MinJobAge. > -- Fix filtering arguments being ignored when using squeue --json. > -- switch/nvidia_imex - Move setup call after spank_init() to allow namespace > manipulation within the SPANK plugin. > -- switch/nvidia_imex - Skip plugin operation if nvidia-caps-imex-channels > device is not present rather than preventing slurmd from starting. > -- switch/nvidia_imex - Skip plugin operation if job_container/tmpfs > is configured due to incompatibility. > -- switch/nvidia_imex - Remove any pre-existing channels when slurmd starts. > -- rpc_queue - Add support for an optional rpc_queue.yaml configuration file.

1 0

SLURM QoS Preemption Not Functioning as Expected
by 木子 27 Jun '24

27 Jun '24

Hi all; I'm testing slurm preempt with PreemptType=preempt/qos and PreemptMode=suspend,gang in slurm.conf. But job with high-qos not preempt the job with low-qos. My configuration followed: slurm version: 23.11.4 test node : 8 CPUS and 8GB RAM QoS Configuration:  Partition Configuration:  Job Submission: I submitted test jobs, one with --qos=low-qos and--partition=low , resources 4 CPUs, 4GB memory, and later, a  job  with --qos=high-qos and  --partition=g1 requesting more resources( 5 CPUs, 5GB memory). Did anyone else encounter this issue or any additional steps I might need to take to correctly enable QoS-based preemption in our SLURM setup. Best regards, Liyar

1 0

2025

2024