- slurm-users - lists.schedmd.com

slurmd error: port already in use, resulting in slaves not being able to communicate with master slurmctld
by Shooktija S N 30 Jul '24

30 Jul '24

Hi, I'm trying to set up a Slurm (version 22.05.8) cluster consisting of 3 nodes with these hostnames and local IP addresses: server1 - 10.36.17.152 server2 - 10.36.17.166 server3 - 10.36.17.132 I had scrambled together a minimum working example using these resources: https://github.com/SergioMEV/slurm-for-dummies https://blog.devops.dev/slurm-complete-guide-a-to-z-concepts-setup-and-trou… For a while everything looked fine and I was able to run the command I usually use to see if everything is fine: srun --label --nodes=3 hostname Which used to show the expected output of the hostnames of all 3 computers, namely: server1, server2, and server3. However - after having made no changes to the configs - the command no longer works if I specify the number of nodes as anything more than 1, this behaviour is consistent on all 3 computers, the output of 'sinfo' is also included below: root@server1:~# srun --nodes=1 hostnameserver1root@server1:~# root@server1:~# srun --nodes=3 hostnamesrun: Required node not available (down, drained or reserved)srun: job 312 queued and waiting for resources^Csrun: Job allocation 312 has been revokedsrun: Force Terminated JobId=312root@server1:~# root@server1:~# ssh server2 "srun --nodes=1 hostname"server1root@server1:~# root@server1:~# ssh server2 "srun --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 314 queued and waiting for resources^Croot@server1:~# root@server1:~# root@server1:~# sinfoPARTITION AVAIL TIMELIMIT NODES STATE NODELISTmainPartition* up infinite 2 down* server[2-3]mainPartition* up infinite 1 idle server1root@server1:~# Turns out, slurmctld on the master node (hostname: server1) and slurmd on the slave nodes (hostnames: server2 & server3) are throwing some errors probably related to networking: A few lines before and after the first occurence of the error in slurmctld.log on the master node - it's the only type of error I have noticed in the logs (pastebin to the entire log <https://pastebin.com/GBSWXZJR>): root@server1:/var/log# grep -B 20 -A 5 -m1 -i "error" slurmctld.log[2024-07-26T13:13:49.579] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions[2024-07-26T13:13:49.580] debug: power_save module disabled, SuspendTime < 0[2024-07-26T13:13:49.580] Running as primary controller[2024-07-26T13:13:49.580] debug: No backup controllers, not launching heartbeat.[2024-07-26T13:13:49.580] debug: priority/basic: init: Priority BASIC plugin loaded[2024-07-26T13:13:49.580] No parameter for mcs plugin, default values set[2024-07-26T13:13:49.580] mcs: MCSParameters = (null). ondemand set.[2024-07-26T13:13:49.580] debug: mcs/none: init: mcs none plugin loaded[2024-07-26T13:13:49.580] debug2: slurmctld listening on 0.0.0.0:6817[2024-07-26T13:13:52.662] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded[2024-07-26T13:13:52.662] debug2: Processing RPC: MESSAGE_NODE_REGISTRATION_STATUS from UID=0[2024-07-26T13:13:52.662] debug: gres/gpu: init: loaded[2024-07-26T13:13:52.662] debug: validate_node_specs: node server1 registered with 0 jobs[2024-07-26T13:13:52.662] debug2: _slurm_rpc_node_registration complete for server1 usec=229[2024-07-26T13:13:53.586] debug: Spawning registration agent for server[2-3] 2 hosts[2024-07-26T13:13:53.586] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2[2024-07-26T13:13:53.586] debug: sched: Running job scheduler for default depth.[2024-07-26T13:13:53.586] debug2: Spawning RPC agent for msg_type REQUEST_NODE_REGISTRATION_STATUS[2024-07-26T13:13:53.587] debug2: Tree head got back 0 looking for 2[2024-07-26T13:13:53.588] debug2: _slurm_connect: failed to connect to 10.36.17.166:6818: Connection refused[2024-07-26T13:13:53.588] debug2: Error connecting slurm stream socket at 10.36.17.166:6818: Connection refused[2024-07-26T13:13:53.588] debug2: _slurm_connect: failed to connect to 10.36.17.132:6818: Connection refused[2024-07-26T13:13:53.588] debug2: Error connecting slurm stream socket at 10.36.17.132:6818: Connection refused[2024-07-26T13:13:54.588] debug2: _slurm_connect: failed to connect to 10.36.17.166:6818: Connection refused[2024-07-26T13:13:54.588] debug2: Error connecting slurm stream socket at 10.36.17.166:6818: Connection refused[2024-07-26T13:13:54.589] debug2: _slurm_connect: failed to connect to 10.36.17.132:6818: Connection refused The connections to 10.36.17.166:6818 and 10.36.17.132:6818 are refused. Those are ports specified by the 'SlurmdPort' key in slurm.conf There are similar errors in the slurmd.log files on both the slave nodes as well: slurmd.log on server2, the error is only at the end of the file (pastebin to the entire log <https://pastebin.com/TwSMiAp7>): root@server2:/var/log# tail -5 slurmd.log [2024-07-26T13:13:53.018] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.018] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.018] debug2: No mpi.conf file (/etc/slurm/mpi.conf)[2024-07-26T13:13:53.018] error: Error binding slurm stream socket: Address already in use[2024-07-26T13:13:53.018] error: Unable to bind listen port (6818): Address already in use slurmd.log on server3 (pastebin to the entire log <https://pastebin.com/K55cAGLb>): root@server3:/var/log# tail -5 slurmd.log [2024-07-26T13:13:53.383] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.383] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.383] debug2: No mpi.conf file (/etc/slurm/mpi.conf)[2024-07-26T13:13:53.384] error: Error binding slurm stream socket: Address already in use[2024-07-26T13:13:53.384] error: Unable to bind listen port (6818): Address already in use I use this script to restart slurm whenever I change any of the configs, could the order in which these operations are being done cause the problems I'm facing: #! /bin/bashscp /etc/slurm/slurm.conf /etc/slurm/gres.conf server2:/etc/slurm/ && echo copied slurm.conf and gres.conf to server2;scp /etc/slurm/slurm.conf /etc/slurm/gres.conf server3:/etc/slurm/ && echo copied slurm.conf and gres.conf to server3;echoecho restarting slurmctld and slurmd on server1(scontrol shutdown ; sleep 3 ; rm -f /var/log/slurmd.log /var/log/slurmctld.log ; slurmctld -d ; sleep 3 ; slurmd) && echo doneecho restarting slurmd on server2(ssh server2 "rm -f /var/log/slurmd.log /var/log/slurmctld.log ; slurmd") && echo doneecho restarting slurmd on server3(ssh server3 "rm -f /var/log/slurmd.log /var/log/slurmctld.log ; slurmd") && echo done Config files: slurm.conf without the comments: root@server1:/etc/slurm# grep -v "#" slurm.conf ClusterName=DlabClusterSlurmctldHost=server1GresTypes=gpuProctrackType=proctrack/linuxprocReturnToService=1SlurmctldPidFile=/var/run/slurmctld.pidSlurmctldPort=6817SlurmdPidFile=/var/run/slurmd.pidSlurmdPort=6818SlurmdSpoolDir=/var/spool/slurmdSlurmUser=rootStateSaveLocation=/var/spool/slurmctldTaskPlugin=task/affinity,task/cgroupInactiveLimit=0KillWait=30MinJobAge=300SlurmctldTimeout=120SlurmdTimeout=300Waittime=0SchedulerType=sched/backfillSelectType=select/cons_tresJobCompType=jobcomp/noneJobAcctGatherFrequency=30SlurmctldDebug=debug2SlurmctldLogFile=/var/log/slurmctld.logSlurmdDebug=debug2SlurmdLogFile=/var/log/slurmd.logNodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP gres.conf: root@server1:/etc/slurm# cat gres.confNodeName=server1 Name=gpu File=/dev/nvidia0NodeName=server2 Name=gpu File=/dev/nvidia0NodeName=server3 Name=gpu File=/dev/nvidia0 These config files are the same on all 3 computers. As a complete beginner to Linux and Slurm administration, I have been struggling to understand even the most basic documentation, and I have been unable to find answers online. Any assistance would be greatly appreciated. Thanks!

1 1

Final Call for SLUG Standard Registration
by Victoria Hobson 29 Jul '24

29 Jul '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information, abstracts, and travel recommendations can be found here:https://slug24.splashthat.com/ The last day to register with standard pricing ($900) is this Friday, August 2nd. After this, final registration will run until August 30th at a price of $1100. SLUG is the best way to interact with the Slurm community and to interact with the SchedMD Support & Training staff. Don't forget to register. We can't wait to see you in Oslo! -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

Convergence of Kube and Slurm?
by Dan Healy 29 Jul '24

29 Jul '24

Bright Cluster Manager has some verbiage on their marketing site that they can manage a cluster running both Kubernetes and Slurm. Maybe I misunderstood it. But nevertheless, I am encountering groups more frequently that want to run a stack of containers that need private container networking. What’s the current state of using the same HPC cluster for both Slurm and Kube? Note: I’m aware that I can run Kube on a single node, but we need more resources. So ultimately we need a way to have Slurm and Kube exist in the same cluster, both sharing the full amount of resources and both being fully aware of resource usage. Thanks, Daniel Healy

5 4

Slurm fails before nvidia-smi command
by Aziz Ogutlu 29 Jul '24

29 Jul '24

Hi there all, We have Dell server with 2 x Nvidia H100 and running slurm on it. After restart server if we do not write nvidia-smi command slurm fails. When we run nvidia-smi && systemctl restart slurmd && systemctl restart slurmctld , slurm queue begins. Do you have any idea about this error and what can we do for this issue? -- Best regards, Aziz Öğütlü Eduline Bilişim Sanayi ve Ticaret Ltd. Şti. www.eduline.com.tr Merkez Mah. Ayazma Cad. No:37 Papirus Plaza Kat:6 Ofis No:118 Kağıthane - İstanbul - Türkiye 34406 Tel : +90 212 324 60 61 Cep: +90 541 350 40 72

4 4

Slurm sacct ResvCPURAW invalid field in version 24.12.5
by Mike Mikailov 29 Jul '24

29 Jul '24

Dear All, Does anyone know what is the equivalent field for ResvCPURAW of sacct command of Slurm in its version 23.11.5. I get: sacct: error: Invalid field requested “ResvCPURAW”. Your help at your earliest is very much appreciated. Best, -Mike Sent from my iPhone

2 1

Background tasks in Slurm scripts?
by Steffen Grunewald 26 Jul '24

26 Jul '24

Good morning, yesterday I came across a Slurm (sbatch) script that, after doing some stuff in the foreground, runs another executable in the background - and doesn't "wait" for it to finish - literally the last line of the script is executable & (and that executable is supposed to take several 10 seconds or more to finish) How would Slurm handle this? Will the end of the script immediately trigger the job epilog, and what would happen to the leftover task? This certainly is discussed somewhere in the manual pages and other documentation but up to now I failed to find that place... Thanks, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

3 3

API - Specify GPUs
by jpuerto＠psc.edu 26 Jul '24

26 Jul '24

Good Afternoon All, It does not seem that the REST API allows for folks to configure their jobs to utilize GPUs, using the traditional methods. IE, there does not appear to be an equivalent between the --gpus (or --gres) flag on sbatch/srun and the REST API's job submission endpoint. Can anyone point me towards what should be used if the version we are on does not support tres specifications? Best regards, Juan Puerto

2 1

Error binding slurm stream socket: Address already in use, and GPU GRES verification
by Shooktija S N 23 Jul '24

23 Jul '24

Hi, I am trying to set up Slurm with GPUs as GRES on a 3 node configuration (hostnames: server1, server2, server3). For a while everything looked fine and I was able to run srun --label --nodes=3 hostname which is what I use to test if Slurm is working correctly, and then it randomly stops. Turns out slurmctld is not working and it throws the following error (the two lines are consecutive in the log file): root@server1:/var/log# grep -i error slurmctld.log [2024-07-22T14:47:32.302] error: Error binding slurm stream socket: Address already in use[2024-07-22T14:47:32.302] fatal: slurm_init_msg_engine_port error Address already in use This error is being thrown after having made no changes to the config files, in fact the cluster wasn't used at all for a few weeks before this error was thrown. This is the simple script I use to restart Slurm: root@server1:~# cat slurmRestart.sh #! /bin/bashscp /etc/slurm/slurm.conf server2:/etc/slurm/ && echo copied slurm.conf to server2;scp /etc/slurm/slurm.conf server3:/etc/slurm/ && echo copied slurm.conf to server3;rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld ; echo restarting slurm on server1;(ssh server2 "rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld") && echo restarting slurm on server2;(ssh server3 "rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld") && echo restarting slurm on server3; Could the error be due to the slurmd and/or slurmctld not being started in the right order? The other question I have is regarding the configuration of a GPU as a GRES - how do I verify that it has been configured correctly? I was told to use srun nvidia-smi with and without having enabled GPU use, but whether or not I enable GPU usage has no effect on the output of the command: root@server1:~# srun --nodes=1 nvidia-smi --query-gpu=uuid --format=csvuuidGPU-55f127a8-dbf4-fd12-3cad-c0d5f2dcb005root@server1:~# root@server1:~# srun --nodes=1 --gpus-per-node=1 nvidia-smi --query-gpu=uuid --format=csvuuidGPU-55f127a8-dbf4-fd12-3cad-c0d5f2dcb005 I am sceptical if about whether the GPU has properly been configured, is this the best way to check if it has? *The error:* I first noticed this happening when I tried to run the command I usually use to see if everything is fine, the srun command runs only one node, and the only way to stop it if I specify the number of nodes as 3 is to press Ctrl+C: root@server1:~# srun --label --nodes=1 hostname0: server1root@server1:~# ssh server2 "srun --label --nodes=1 hostname"0: server1root@server1:~# ssh server3 "srun --label --nodes=1 hostname"0: server1root@server1:~# srun --label --nodes=3 hostnamesrun: Required node not available (down, drained or reserved)srun: job 265 queued and waiting for resources^Csrun: Job allocation 265 has been revokedsrun: Force Terminated JobId=265root@server1:~# ssh server2 "srun --label --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 266 queued and waiting for resources^Croot@server1:~# ssh server3 "srun --label --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 267 queued and waiting for resourcesroot@server1:~# *The logs:* 1) The last 30 lines of */var/log/slurmctld.log* at the debug5 level in server #1 (pastebin to the entire log <https://pastebin.com/fw4C4xtr>): root@server1:/var/log# tail -30 slurmctld.log [2024-07-22T14:47:32.301] debug: Updating partition uid access list[2024-07-22T14:47:32.301] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/resv_state` as buf_t[2024-07-22T14:47:32.301] debug3: Version string in resv_state header is PROTOCOL_VERSION[2024-07-22T14:47:32.301] Recovered state of 0 reservations[2024-07-22T14:47:32.301] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/trigger_state` as buf_t[2024-07-22T14:47:32.301] State of 0 triggers recovered[2024-07-22T14:47:32.301] read_slurm_conf: backup_controller not specified[2024-07-22T14:47:32.301] select/cons_tres: select_p_reconfigure: select/cons_tres: reconfigure[2024-07-22T14:47:32.301] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions[2024-07-22T14:47:32.301] debug: power_save module disabled, SuspendTime < 0[2024-07-22T14:47:32.301] Running as primary controller[2024-07-22T14:47:32.301] debug: No backup controllers, not launching heartbeat.[2024-07-22T14:47:32.301] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/priority_basic.so[2024-07-22T14:47:32.301] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Priority BASIC plugin type:priority/basic version:0x160508[2024-07-22T14:47:32.301] debug: priority/basic: init: Priority BASIC plugin loaded[2024-07-22T14:47:32.301] debug3: Success.[2024-07-22T14:47:32.301] No parameter for mcs plugin, default values set[2024-07-22T14:47:32.301] mcs: MCSParameters = (null). ondemand set.[2024-07-22T14:47:32.301] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/mcs_none.so[2024-07-22T14:47:32.301] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:mcs none plugin type:mcs/none version:0x160508[2024-07-22T14:47:32.301] debug: mcs/none: init: mcs none plugin loaded[2024-07-22T14:47:32.301] debug3: Success.[2024-07-22T14:47:32.302] debug3: _slurmctld_rpc_mgr pid = 3159324[2024-07-22T14:47:32.302] debug3: _slurmctld_background pid = 3159324[2024-07-22T14:47:32.302] error: Error binding slurm stream socket: Address already in use[2024-07-22T14:47:32.302] fatal: slurm_init_msg_engine_port error Address already in use[2024-07-22T14:47:32.304] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.304] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.304] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.304] slurmscriptd: debug: _slurmscriptd_mainloop: finished 2) Entirety of *slurmctld.log on server #2*: root@server2:/var/log# cat slurmctld.log [2024-07-22T14:47:32.614] debug: slurmctld log levels: stderr=debug5 logfile=debug5 syslog=quiet[2024-07-22T14:47:32.614] debug: Log file re-opened[2024-07-22T14:47:32.615] slurmscriptd: debug: slurmscriptd: Got ack from slurmctld, initialization successful[2024-07-22T14:47:32.615] slurmscriptd: debug: _slurmscriptd_mainloop: started[2024-07-22T14:47:32.616] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.616] debug: slurmctld: slurmscriptd fork()'d and initialized.[2024-07-22T14:47:32.616] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.616] debug: _slurmctld_listener_thread: started listening to slurmscriptd[2024-07-22T14:47:32.616] debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.616] debug3: Called _msg_readable[2024-07-22T14:47:32.616] slurmctld version 22.05.8 started on cluster dlabcluster[2024-07-22T14:47:32.616] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/cred_munge.so[2024-07-22T14:47:32.616] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x160508[2024-07-22T14:47:32.616] cred/munge: init: Munge credential signature plugin loaded[2024-07-22T14:47:32.616] debug3: Success.[2024-07-22T14:47:32.616] error: This host (server2/server2) not a valid controller[2024-07-22T14:47:32.617] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.617] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.617] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.617] slurmscriptd: debug: _slurmscriptd_mainloop: finished 3) Entirety of *slurmctld.log on server #3*: root@server3:/var/log# cat slurmctld.log [2024-07-22T14:47:32.927] debug: slurmctld log levels: stderr=debug5 logfile=debug5 syslog=quiet[2024-07-22T14:47:32.927] debug: Log file re-opened[2024-07-22T14:47:32.928] slurmscriptd: debug: slurmscriptd: Got ack from slurmctld, initialization successful[2024-07-22T14:47:32.928] slurmscriptd: debug: _slurmscriptd_mainloop: started[2024-07-22T14:47:32.928] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.928] debug: slurmctld: slurmscriptd fork()'d and initialized.[2024-07-22T14:47:32.928] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.928] slurmctld version 22.05.8 started on cluster dlabcluster[2024-07-22T14:47:32.929] debug: _slurmctld_listener_thread: started listening to slurmscriptd[2024-07-22T14:47:32.929] debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.929] debug3: Called _msg_readable[2024-07-22T14:47:32.929] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/cred_munge.so[2024-07-22T14:47:32.929] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x160508[2024-07-22T14:47:32.929] cred/munge: init: Munge credential signature plugin loaded[2024-07-22T14:47:32.929] debug3: Success.[2024-07-22T14:47:32.929] error: This host (server3/server3) not a valid controller[2024-07-22T14:47:32.930] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.930] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.930] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.930] slurmscriptd: debug: _slurmscriptd_mainloop: finished *The config files (shared by all 3 computers):* 1) */etc/slurm/slurm.conf* without the comments: root@server1:/etc/slurm# grep -v "#" slurm.conf ClusterName=DlabClusterSlurmctldHost=server1GresTypes=gpuProctrackType=proctrack/linuxprocReturnToService=1SlurmctldPidFile=/var/run/slurmctld.pidSlurmctldPort=6817SlurmdPidFile=/var/run/slurmd.pidSlurmdPort=6818SlurmdSpoolDir=/var/spool/slurmdSlurmUser=rootStateSaveLocation=/var/spool/slurmctldTaskPlugin=task/affinity,task/cgroupInactiveLimit=0KillWait=30MinJobAge=300SlurmctldTimeout=120SlurmdTimeout=300Waittime=0SchedulerType=sched/backfillSelectType=select/cons_tresJobCompType=jobcomp/noneJobAcctGatherFrequency=30SlurmctldDebug=debug5SlurmctldLogFile=/var/log/slurmctld.logSlurmdDebug=debug5SlurmdLogFile=/var/log/slurmd.logNodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP 2) */etc/slurm/gres.conf*: root@server1:/etc/slurm# cat gres.conf NodeName=server1 Name=gpu File=/dev/nvidia0NodeName=server2 Name=gpu File=/dev/nvidia0NodeName=server3 Name=gpu File=/dev/nvidia0 These files are the same on all 3 computers: root@server1:/etc/slurm# diff slurm.conf <(ssh server2 "cat /etc/slurm/slurm.conf")root@server1:/etc/slurm# diff slurm.conf <(ssh server3 "cat /etc/slurm/slurm.conf")root@server1:/etc/slurm# diff gres.conf <(ssh server2 "cat /etc/slurm/gres.conf")root@server1:/etc/slurm# diff gres.conf <(ssh server3 "cat /etc/slurm/gres.conf")root@server1:/etc/slurm# Thank you, Shookti

1 0

Cgroup
by stth 22 Jul '24

22 Jul '24

Hello, I am configuring cgroups on my server for the first time. I've created a cgroup.conf file in the Slurm directory with the following values: ConstrainCores=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes AllowedRAMSpace=90 AllowedSwapSpace=10 I feel like this configuration might be incomplete. Can anyone provide assistance? I haven't found any specific guidance online. Thanks!

2 1

Custom Plugin Integration
by Bhaskar Chakraborty 20 Jul '24

20 Jul '24

Hello, We wish to have a schedulingintegration with Slurm. Our own application has a backend system which willdecide the placement of jobs across hosts & CPU cores. The backend takesits own time to come back with a placement (which may take a few seconds) & we expect slurm to update it regularly about any change in the current state ofavailable resources. For this we believe we have 3 options broadly: - We use the const_tres Select plugin & modify it to let it query our backend system for job placements. - We write our own Select plugin avoiding any other Select plugin. - We use existing select plugin & also register our own plugin. Idea is that our plugin will cater to 'our' jobs (specific partition, say) while all other jobs would be taken up by the default plugin. Problem with a> is that this leads to modification ofexisting plugin code & calling (our) library code from inside Select pluginlib. With b> the issue is unless we have the full Slurm cluster toourselves this isn't viable. Any insight how to proceed with this? Where shouldour select plugin, assuming we need to make one, fits in the slurm integration. We are not sure whether c> is allowed in Slurm. We went through existing Select plugins Linear & cons_tres.However, not able to figure out how to use them or write something on similarlines to suit our purpose. Any help in this regard is appreciated. Apologies if this question (or any other very similar) is already answered,please point to the relevant thread then. Thanks in advance for any pointers. Regards, Bhaskar.

3 7