- slurm-users - lists.schedmd.com

Error binding slurm stream socket: Address already in use, and GPU GRES verification
by Shooktija S N 23 Jul '24

23 Jul '24

Hi, I am trying to set up Slurm with GPUs as GRES on a 3 node configuration (hostnames: server1, server2, server3). For a while everything looked fine and I was able to run srun --label --nodes=3 hostname which is what I use to test if Slurm is working correctly, and then it randomly stops. Turns out slurmctld is not working and it throws the following error (the two lines are consecutive in the log file): root@server1:/var/log# grep -i error slurmctld.log [2024-07-22T14:47:32.302] error: Error binding slurm stream socket: Address already in use[2024-07-22T14:47:32.302] fatal: slurm_init_msg_engine_port error Address already in use This error is being thrown after having made no changes to the config files, in fact the cluster wasn't used at all for a few weeks before this error was thrown. This is the simple script I use to restart Slurm: root@server1:~# cat slurmRestart.sh #! /bin/bashscp /etc/slurm/slurm.conf server2:/etc/slurm/ && echo copied slurm.conf to server2;scp /etc/slurm/slurm.conf server3:/etc/slurm/ && echo copied slurm.conf to server3;rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld ; echo restarting slurm on server1;(ssh server2 "rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld") && echo restarting slurm on server2;(ssh server3 "rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld") && echo restarting slurm on server3; Could the error be due to the slurmd and/or slurmctld not being started in the right order? The other question I have is regarding the configuration of a GPU as a GRES - how do I verify that it has been configured correctly? I was told to use srun nvidia-smi with and without having enabled GPU use, but whether or not I enable GPU usage has no effect on the output of the command: root@server1:~# srun --nodes=1 nvidia-smi --query-gpu=uuid --format=csvuuidGPU-55f127a8-dbf4-fd12-3cad-c0d5f2dcb005root@server1:~# root@server1:~# srun --nodes=1 --gpus-per-node=1 nvidia-smi --query-gpu=uuid --format=csvuuidGPU-55f127a8-dbf4-fd12-3cad-c0d5f2dcb005 I am sceptical if about whether the GPU has properly been configured, is this the best way to check if it has? *The error:* I first noticed this happening when I tried to run the command I usually use to see if everything is fine, the srun command runs only one node, and the only way to stop it if I specify the number of nodes as 3 is to press Ctrl+C: root@server1:~# srun --label --nodes=1 hostname0: server1root@server1:~# ssh server2 "srun --label --nodes=1 hostname"0: server1root@server1:~# ssh server3 "srun --label --nodes=1 hostname"0: server1root@server1:~# srun --label --nodes=3 hostnamesrun: Required node not available (down, drained or reserved)srun: job 265 queued and waiting for resources^Csrun: Job allocation 265 has been revokedsrun: Force Terminated JobId=265root@server1:~# ssh server2 "srun --label --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 266 queued and waiting for resources^Croot@server1:~# ssh server3 "srun --label --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 267 queued and waiting for resourcesroot@server1:~# *The logs:* 1) The last 30 lines of */var/log/slurmctld.log* at the debug5 level in server #1 (pastebin to the entire log <https://pastebin.com/fw4C4xtr>): root@server1:/var/log# tail -30 slurmctld.log [2024-07-22T14:47:32.301] debug: Updating partition uid access list[2024-07-22T14:47:32.301] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/resv_state` as buf_t[2024-07-22T14:47:32.301] debug3: Version string in resv_state header is PROTOCOL_VERSION[2024-07-22T14:47:32.301] Recovered state of 0 reservations[2024-07-22T14:47:32.301] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/trigger_state` as buf_t[2024-07-22T14:47:32.301] State of 0 triggers recovered[2024-07-22T14:47:32.301] read_slurm_conf: backup_controller not specified[2024-07-22T14:47:32.301] select/cons_tres: select_p_reconfigure: select/cons_tres: reconfigure[2024-07-22T14:47:32.301] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions[2024-07-22T14:47:32.301] debug: power_save module disabled, SuspendTime < 0[2024-07-22T14:47:32.301] Running as primary controller[2024-07-22T14:47:32.301] debug: No backup controllers, not launching heartbeat.[2024-07-22T14:47:32.301] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/priority_basic.so[2024-07-22T14:47:32.301] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Priority BASIC plugin type:priority/basic version:0x160508[2024-07-22T14:47:32.301] debug: priority/basic: init: Priority BASIC plugin loaded[2024-07-22T14:47:32.301] debug3: Success.[2024-07-22T14:47:32.301] No parameter for mcs plugin, default values set[2024-07-22T14:47:32.301] mcs: MCSParameters = (null). ondemand set.[2024-07-22T14:47:32.301] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/mcs_none.so[2024-07-22T14:47:32.301] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:mcs none plugin type:mcs/none version:0x160508[2024-07-22T14:47:32.301] debug: mcs/none: init: mcs none plugin loaded[2024-07-22T14:47:32.301] debug3: Success.[2024-07-22T14:47:32.302] debug3: _slurmctld_rpc_mgr pid = 3159324[2024-07-22T14:47:32.302] debug3: _slurmctld_background pid = 3159324[2024-07-22T14:47:32.302] error: Error binding slurm stream socket: Address already in use[2024-07-22T14:47:32.302] fatal: slurm_init_msg_engine_port error Address already in use[2024-07-22T14:47:32.304] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.304] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.304] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.304] slurmscriptd: debug: _slurmscriptd_mainloop: finished 2) Entirety of *slurmctld.log on server #2*: root@server2:/var/log# cat slurmctld.log [2024-07-22T14:47:32.614] debug: slurmctld log levels: stderr=debug5 logfile=debug5 syslog=quiet[2024-07-22T14:47:32.614] debug: Log file re-opened[2024-07-22T14:47:32.615] slurmscriptd: debug: slurmscriptd: Got ack from slurmctld, initialization successful[2024-07-22T14:47:32.615] slurmscriptd: debug: _slurmscriptd_mainloop: started[2024-07-22T14:47:32.616] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.616] debug: slurmctld: slurmscriptd fork()'d and initialized.[2024-07-22T14:47:32.616] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.616] debug: _slurmctld_listener_thread: started listening to slurmscriptd[2024-07-22T14:47:32.616] debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.616] debug3: Called _msg_readable[2024-07-22T14:47:32.616] slurmctld version 22.05.8 started on cluster dlabcluster[2024-07-22T14:47:32.616] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/cred_munge.so[2024-07-22T14:47:32.616] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x160508[2024-07-22T14:47:32.616] cred/munge: init: Munge credential signature plugin loaded[2024-07-22T14:47:32.616] debug3: Success.[2024-07-22T14:47:32.616] error: This host (server2/server2) not a valid controller[2024-07-22T14:47:32.617] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.617] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.617] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.617] slurmscriptd: debug: _slurmscriptd_mainloop: finished 3) Entirety of *slurmctld.log on server #3*: root@server3:/var/log# cat slurmctld.log [2024-07-22T14:47:32.927] debug: slurmctld log levels: stderr=debug5 logfile=debug5 syslog=quiet[2024-07-22T14:47:32.927] debug: Log file re-opened[2024-07-22T14:47:32.928] slurmscriptd: debug: slurmscriptd: Got ack from slurmctld, initialization successful[2024-07-22T14:47:32.928] slurmscriptd: debug: _slurmscriptd_mainloop: started[2024-07-22T14:47:32.928] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.928] debug: slurmctld: slurmscriptd fork()'d and initialized.[2024-07-22T14:47:32.928] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.928] slurmctld version 22.05.8 started on cluster dlabcluster[2024-07-22T14:47:32.929] debug: _slurmctld_listener_thread: started listening to slurmscriptd[2024-07-22T14:47:32.929] debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.929] debug3: Called _msg_readable[2024-07-22T14:47:32.929] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/cred_munge.so[2024-07-22T14:47:32.929] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x160508[2024-07-22T14:47:32.929] cred/munge: init: Munge credential signature plugin loaded[2024-07-22T14:47:32.929] debug3: Success.[2024-07-22T14:47:32.929] error: This host (server3/server3) not a valid controller[2024-07-22T14:47:32.930] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.930] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.930] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.930] slurmscriptd: debug: _slurmscriptd_mainloop: finished *The config files (shared by all 3 computers):* 1) */etc/slurm/slurm.conf* without the comments: root@server1:/etc/slurm# grep -v "#" slurm.conf ClusterName=DlabClusterSlurmctldHost=server1GresTypes=gpuProctrackType=proctrack/linuxprocReturnToService=1SlurmctldPidFile=/var/run/slurmctld.pidSlurmctldPort=6817SlurmdPidFile=/var/run/slurmd.pidSlurmdPort=6818SlurmdSpoolDir=/var/spool/slurmdSlurmUser=rootStateSaveLocation=/var/spool/slurmctldTaskPlugin=task/affinity,task/cgroupInactiveLimit=0KillWait=30MinJobAge=300SlurmctldTimeout=120SlurmdTimeout=300Waittime=0SchedulerType=sched/backfillSelectType=select/cons_tresJobCompType=jobcomp/noneJobAcctGatherFrequency=30SlurmctldDebug=debug5SlurmctldLogFile=/var/log/slurmctld.logSlurmdDebug=debug5SlurmdLogFile=/var/log/slurmd.logNodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP 2) */etc/slurm/gres.conf*: root@server1:/etc/slurm# cat gres.conf NodeName=server1 Name=gpu File=/dev/nvidia0NodeName=server2 Name=gpu File=/dev/nvidia0NodeName=server3 Name=gpu File=/dev/nvidia0 These files are the same on all 3 computers: root@server1:/etc/slurm# diff slurm.conf <(ssh server2 "cat /etc/slurm/slurm.conf")root@server1:/etc/slurm# diff slurm.conf <(ssh server3 "cat /etc/slurm/slurm.conf")root@server1:/etc/slurm# diff gres.conf <(ssh server2 "cat /etc/slurm/gres.conf")root@server1:/etc/slurm# diff gres.conf <(ssh server3 "cat /etc/slurm/gres.conf")root@server1:/etc/slurm# Thank you, Shookti

1 0

Cgroup
by stth 22 Jul '24

22 Jul '24

Hello, I am configuring cgroups on my server for the first time. I've created a cgroup.conf file in the Slurm directory with the following values: ConstrainCores=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes AllowedRAMSpace=90 AllowedSwapSpace=10 I feel like this configuration might be incomplete. Can anyone provide assistance? I haven't found any specific guidance online. Thanks!

2 1

Custom Plugin Integration
by Bhaskar Chakraborty 21 Jul '24

21 Jul '24

Hello, We wish to have a schedulingintegration with Slurm. Our own application has a backend system which willdecide the placement of jobs across hosts & CPU cores. The backend takesits own time to come back with a placement (which may take a few seconds) & we expect slurm to update it regularly about any change in the current state ofavailable resources. For this we believe we have 3 options broadly: - We use the const_tres Select plugin & modify it to let it query our backend system for job placements. - We write our own Select plugin avoiding any other Select plugin. - We use existing select plugin & also register our own plugin. Idea is that our plugin will cater to 'our' jobs (specific partition, say) while all other jobs would be taken up by the default plugin. Problem with a> is that this leads to modification ofexisting plugin code & calling (our) library code from inside Select pluginlib. With b> the issue is unless we have the full Slurm cluster toourselves this isn't viable. Any insight how to proceed with this? Where shouldour select plugin, assuming we need to make one, fits in the slurm integration. We are not sure whether c> is allowed in Slurm. We went through existing Select plugins Linear & cons_tres.However, not able to figure out how to use them or write something on similarlines to suit our purpose. Any help in this regard is appreciated. Apologies if this question (or any other very similar) is already answered,please point to the relevant thread then. Thanks in advance for any pointers. Regards, Bhaskar.

3 7

CLOUD nodes with unknown IP addresses
by Martin Lee 20 Jul '24

20 Jul '24

I've read the following in the slurm power saving docs: https://slurm.schedmd.com/power_save.html > *cloud_dns* > By default, Slurm expects that the network addresses for cloud nodes won't be known until creation of the node and that Slurm will be notified of the node's address upon registration. Since Slurm communications rely on the node configuration found in the slurm.conf, Slurm will tell the client command, after waiting for all nodes to boot, each node's IP address. However, in environments where the nodes are in DNS, this step can be avoided by configuring this option. > I am creating the nodes on demand and don't know the IP ahead of the instance start, so cloud_dns is not set. I'm confused specifically by "Slurm will be notified of the node's address upon registration." Who/what is expected to do this? If it is expected to be performed by the ResumeProgram, does it need to be done before slurmd starts on the node? Is it OK if the node does it after slurmd has started with something like: scontrol update nodename=$(hostname -s) nodeaddr=$(hostname -I) nodehostname=$(hostname) scontrol reconfigure Thank you, Martin

2 2

Slurm Users cannot open gui login nodes with X11 without logging into the Compute node.
by Alper AYKUT 18 Jul '24

18 Jul '24

Hello 1 Login Server I have a total of 3 calculation nodes. Especially in non-GUI jobs, I can run scripts successfully with Slurm sbatch. However, when it is with Gui, I run it with X11. However, my problem for now is this: When I run sbatch gui.sh, it runs smoothly and automatically selects the calculation node and gui is active on the login server. But the node it selects must be logged in with the same user for x11 to work. How can I run the application with gui on the login node without logging in on the calculation node? The calculation node gives the following error when I run the script without logging in. How can I solve this problem. srun: error: Ignoring --x11 option for a job step within an existing job. Set x11 options at job allocation time. srun: error: tutelservernode01: task 0: Exited with exit code 255 My test Script ; #!/bin/bash #SBATCH --partition=debug #Kullanılacak partition (compute, debug, vb.) #SBATCH --job-name=xclock # İşin adı #SBATCH --output=xclock-%j.log # İşin çıktı dosyası #SBATCH --error=xclock-%j.err # İşin hata çıktı dosyası # Slurm'un hangi node'dan işlemi çalıştıracağını seçmesine izin verin # --nodelist parametresi kullanılmazsa, Slurm otomatik olarak bir node seçecektir. # xclock komutunu başlatmak için DISPLAY ortam değişkenini ayarlayın #export DISPLAY=localhost:10.0 # MobaXterm veya başka bir X sunucusunun adresine göre ayarlayın export DISPLAY=$DISPLAY # xclock komutunu sbatch içinde başlatın srun --x11 xclock thankyou for support.

1 0

Nodes TRES double what is requested
by jack.mellor＠mbda.co.uk 12 Jul '24

12 Jul '24

Hi, We are running slurm 23.02.6. Our nodes have hyperthreading disabled and we have slurm.conf set to CPU=32 for each node (each node has 2 processes with 16 cores). When we allocated a job, such as salloc -n 32, it will allocate a whole node but using sinfo shows double the allocation in the TRES=64. It also shows in sinfo that the node has 4294967264 idle CPUs. Not sure if its a known bug, or an issue with our config? I have tried various things, like setting the sockets/boards in slurm.conf. Thanks Jack

5 4

SLURM noob administrator question
by Cutts, Tim 12 Jul '24

12 Jul '24

Still learning about SLURM, so please forgive me if I ask a naïve question I like to use Anders Halager’s gnodes command to visualise the state of our nodes. I’ve noticed lately that we fairly often see things like this (apologies for line wrap): +- core - 46 cores & 186GB -----------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+ | seskscpn301 0G ___OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO | seskscpn309 172G ..........................................!!!! | seskscpn317 0G OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO | Now, you can see in this that nodes 301 and 317 are more or less fully loaded. This is great. But 309 is in an interesting state. Four overloaded cores, and all other cores unused, and plenty of RAM available. And yet SLURM is not scheduling any more work to that node. Right now there are more than 2000 jobs pending, many of which could run on that node. But SLURM is not scheduling them, and I don’t know why. One thing I’ve seen cause this is a job trying to use more CPUs than it has been allocated. The cgroup stops this being a real problem of course, but it does cause the load average to go high. Is this what’s causing SLURM to stop sending anything to the node? Is there a configuration change that might help in this situation? Thanks in advance, Tim -- Tim Cutts Scientific Computing Platform Lead AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>

2 1

Replacing MUNGE with SACK (auth/slurm)
by Daniel Letai 12 Jul '24

12 Jul '24

Does SACK replace MUNGE? As in - MUNGE is not required when building Slurm or on compute? If so, can the Requires and BuildRequires for munge be made optional on bcond_without_munge in the spec file? Or is there a reason MUNGE must remain a hard require for Slurm? Thanks, --Dani_L.

1 0

Job submitted to multiple partitions not running when any partition is full
by Paul Raines 10 Jul '24

10 Jul '24

I have a job 465072 submitted to multiple partitions (rtx6000,rtx8000,pubgpu) JOBID PARTITION PENDING PRIORITY TRES_ALLOC|REASON 4650727 rtx6000 47970 0.00367972 cpu=5,mem=400G,node=1,gpu=1|Priority 4650727 rtx8000 47970 0.00367972 cpu=5,mem=400G,node=1,gpu=1|Priority 4650727 pubgpu 47970 0.00367972 cpu=5,mem=400G,node=1,gpu=1|Priority 4646926 rtx6000 487048 0.00121987 cpu=10,mem=32G,node=1,gpu=1|Priority,Resources 4650186 rtx8000 56979 0.00000000 cpu=4,mem=10G,node=1,gpu=1|Priority,Resources We see the two partitions rtx6000 and rtx8000 are full and two other jobs are at the top of the queue waiting to run on those. But partition pubgpu is NOT full and you can see here a node leo with resources to run the 4650727 job HOST PARTITION CORES MEMORY GPUS leo pubgpu 48/ 64 12288/1030994 0/ 1 leo pubcpu 48/ 64 12288/1030994 0/ 1 The node leo is NOT part of the rtx6000 or rtx8000 partitions and there are no other pending jobs waiting on either the pubgpu or pubcpu partition that leo is part of So why is 4650727 not running on the pubgpu partition? --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

2 2

Temporarily bypassing pam_slurm_adopt.so
by Daniel L'Hommedieu 10 Jul '24

10 Jul '24

Hi, all. We have a use case where we need to allow a group of users (members of an LDAP group, which I can easily add to a Linux group) to SSH to a compute node, without disabling pam_slurm_adopt.so. Is there a way to do this? We can add users to the sudo group, which will bypass pam_slurm_adopt.so, but we do not want to grant sudo access to these users. Is there any way to bypass for a group of users, pam_slurm_adopt.so without removing it from /etc/pam.d/sshd, and without adding them to the sudo group? Thanks. Daniel

5 4

2025

2024