- slurm-users - lists.schedmd.com

Why is my job killed when ResumeTimeout is reached instead of it being requeued?
by Xaver Stiensmeier 09 Dec '24

09 Dec '24

Dear Slurm-user list, when a job fails because the node startup fails (cloud scheduling), the job should be re-queued: Resume Timeout Maximum time permitted (in seconds) between when a node resume request is issued and when the node is actually available for use. Nodes which fail to respond in this time frame will be marked DOWN and the jobs scheduled on the node requeued. however, instead of requeuing the job, it is killed. [2024-11-18T10:41:52.003] node bibigrid-worker-wubqboa1z2kkgx0-0 not resumed by ResumeTimeout(1200) - marking down and power_save [2024-11-18T10:41:52.003] Killing JobId=1 on failed node bibigrid-worker-wubqboa1z2kkgx0-0 [2024-11-18T10:41:52.046] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 reason set to: FailedStartup [2024-11-18T10:41:52.046] power down request repeating for node bibigrid-worker-wubqboa1z2kkgx0-0 Our ResumeProgram does not change the state of the underlying workers, I think we should set the nodes to DOWN explicitly if the startup fails given: *ResumeProgram* is unable to restore a node to service with a responding slurmd and an updated BootTime, it should set the node state to DOWN, which will result in a requeue of any job associated with the node - this will happen automatically if the node doesn't register within ResumeTimeout but in any case as we can see in the log the job should be requeued based on it reaching the ResumeTimeout alone. I am unsure why that is not happening. The power down request is sent by the ResumeFailProgram. We have SlurmctldParameters=idle_on_node_suspend activated, but that shouldn't affect Resume, I guess. My Slurm version is slurm 23.11.5 Best regards, Xaver # More context ## Slurmctld from submitting job to failure [2024-11-18T10:21:45.490] sched: _slurm_rpc_allocate_resources JobId=1 NodeList=bibigrid-worker-wubqboa1z2kkgx0-0 usec=1221 [2024-11-18T10:21:45.499] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:21:58.387] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:21:58.387] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:22:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:23:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:23:20.009] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:23:23.003] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:23:23.398] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:23:23.398] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:23:53.398] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:23:53.398] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:24:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:24:21.484] slurmscriptd: error: _run_script: JobId=0 resumeprog exit status 1:0 [2024-11-18T10:25:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:26:02.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:26:02.417] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:26:02.417] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:26:20.007] debug: sched: Running job scheduler for full queue. [2024-11-18T10:26:32.417] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:26:32.417] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:27:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:28:20.003] debug: Updating partition uid access list [2024-11-18T10:28:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:28:20.008] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:29:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:29:22.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:29:22.448] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:29:22.448] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:30:20.007] debug: sched: Running job scheduler for full queue. [2024-11-18T10:31:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:32:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:32:42.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:32:42.478] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:32:42.478] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:33:12.479] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:33:12.479] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:33:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:33:20.010] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:34:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:35:20.007] debug: sched: Running job scheduler for full queue. [2024-11-18T10:36:01.004] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:36:01.504] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:36:01.504] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:36:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:36:31.505] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:36:31.505] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:37:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:38:20.008] debug: Updating partition uid access list [2024-11-18T10:38:20.008] debug: sched: Running job scheduler for full queue. [2024-11-18T10:38:20.017] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:39:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:39:21.003] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:39:21.530] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:39:21.530] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:39:51.531] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:39:51.531] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:40:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:41:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:41:52.003] node bibigrid-worker-wubqboa1z2kkgx0-0 not resumed by ResumeTimeout(1200) - marking down and power_save [2024-11-18T10:41:52.003] Killing JobId=1 on failed node bibigrid-worker-wubqboa1z2kkgx0-0 [2024-11-18T10:41:52.046] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 reason set to: FailedStartup [2024-11-18T10:41:52.046] power down request repeating for node bibigrid-worker-wubqboa1z2kkgx0-0 [2024-11-18T10:41:52.047] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:41:52.549] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:41:52.549] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:41:52.736] _slurm_rpc_complete_job_allocation: JobId=1 error Job/step already completing or completed [2024-11-18T10:41:53.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:41:53.000] debug: sched: Running job scheduler for default depth. [2024-11-18T10:41:53.014] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 reason set to: FailedStartup [2024-11-18T10:41:53.014] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 state set to IDLE

1 1

getting slurm going
by Steven Jones 08 Dec '24

08 Dec '24

What tests can I do to prove that slurm is talking to the nodes pls? regards Steven

3 5

Non-Standard Mail Notification in Job
by Daniel Miliate 05 Dec '24

05 Dec '24

Newbie here, I'm trying to send myself an email notification with a custom body/subject for notification of important events. Is this possible? I've tried a few things in the script below. Thanks in advance and apologies if this is not how this listing works! - Daniel ---- Script ---- .... #SBATCH --mail-user= example(a)example.example #SBATCH --export=ALL #SBATCH --mail-type=ALL # Both of these work which mail which mailx EMAIL="example(a)example.example" # Attempt 1 echo "Test email body" | mail -s "Test Subject" "$EMAIL" # Attempt 2 mail -s "hello" "$EMAIL" <<EOF hello world EOF # Attempt 3 echo "Specific Information" | mail -s "SLURM Job information" $EMAIL ---- End ----

3 2

slurm not running on a warewulf node
by Steven Jones 04 Dec '24

04 Dec '24

Hi, I have set a log creation/location in slurm.conf as, SlurmdLogFile=/var/log/slurm/slurmd.log But it is 0 length. Slurm will not run, what else do I need to do to log why its failing pls? regards Steven

4 6

slurmd on a warwwulf node - not running
by Steven Jones 04 Dec '24

04 Dec '24

Hi, I have munge running on the controller and nodes fine, all tests passed. I have slurmctld running on the controller ok after checking the logs /var/spool/slurmctld was not created which I assume should have happened via the rpm install? Anyway I cant get slurmd to run on the warewulf nodes and cant find any log to check? How to fault find this? regards Steven

3 3

How can I make sure my user have only one job per node (Job array --exclusive=user,)
by Oren 03 Dec '24

03 Dec '24

Hi, I have a cluster of 20-nodes, and I want to run a jobarray on that cluster, but I want each node to get one job per node. When I do the following: #!/bin/bash #SBATCH --job-name=process_images_train # Job name #SBATCH --time=50:00:00 # Time limit hrs:min:sec #SBATCH --tasks=1 #SBATCH --cpus-per-task=4 #SBATCH --mem=50000 #SBATCH --array=0-19# 19 # Job array with 20 jobs (0 to 19) I get 10 jobs in node #1 and 10 jobs in node #20, I want a job in each node. I've tried: #SBATCH --exclusive=user Also #SBATCH --spread-job #SBATCH --distribution=cyclic Nothing changes, node#1 got 10 jobs and node#2 got 10 jobs. Thanks

2 6

multiple conf-server entries for sackd
by Brian Andrus 03 Dec '24

03 Dec '24

Not sure anyone would know, but... If you are running slurm in HA mode (multiple SlurmctldHost entries) is it possible to point sackd to more than one using the --conf-server option? So either specify --conf-server more than once, or have a comma-delimited list of them? The docs are a little light about that. Brian Andrus

1 0

sinfo not listing any partitions
by Kent L. Hanson 02 Dec '24

02 Dec '24

I am doing a new install of slurm 24.05.3 I have all the packages built and installed on headnode and compute node with the same munge.key, slurm.conf, and gres.conf file. I was able to run munge and unmunge commands to test munge successfully. Time is synced with chronyd. I can't seem to find any useful errors in the logs. For some reason when I run sinfo no nodes are listed. I just see the headers for each column. Has anyone seen this or know what a next step of troubleshooting would be? I'm new to this and not sure where to go from here. Thanks for any and all help! The odd output I am seeing [username@headnode ~] sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST (Nothing is output showing status of partition or nodes) Slurm.conf ClusterName=slurmkvasir SlurmctldHost=kadmin2 MpiDefault=none ProctrackType=proctrack/cgroup PrologFlags=contain ReturnToService=2 SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmctldPort=6817 SlurmPidFile=/var/run/slurm/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/cgroup MinJobAge=600 SchedulerType=sched/backfill SelectType=select/cons_tres PriorityType=priority/multifactor AccountingStorageHost=localhost AccountingStoragePass=/var/run/munge/munge.socket.2 AccountingStorageType=accounting_storage/slurmdbd AccountingStorageTRES=gres/gpu,cpu,node JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmLogFile=/var/log/slurm/slurmd.log nodeName=k[001-448] PartitionName=default Nodes=k[001-448] Default=YES MaxTime=INFINITE State=up Slurmctld.log Error: Configured MailProg is invalid Slurmctld version 24.05.3 started on cluster slurmkvasir Accounting_storage/slurmdbd: clusteracct_storage_p_register_ctld: Regisetering slurmctld at port 8617 Error: read_slurm_conf: default partition not set. Revovered state of 448 nodes Down nodes: k[002-448] Recovered information about 0 jobs Revovered state of 0 reservations Read_slurm_conf: backup_controller not specified Select/cons_tres; select_p_reconfigure: select/cons_tres: reconfigure Running as primary controller Slurmd.log Error: Node configuration differs from hardware: CPUS=1:40(hw) Boards=1:1(hw) SocketsPerBoard=1:2(hw) CoresPerSocket=1:20(hw) ThreadsPerCore:1:1(hw) CPU frequency setting not configured for this node Slurmd version 24.05.3started Slurmd started on Wed, 27 Nov 2024 06:51:03 -0700 CPUS=1 Boards=1 Cores=1 Threads=1 Memory=192030 TmpDisk=95201 uptime 166740 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) Error: _forward_thread: failed to k019 (10.142.0.119:6818): Connection timed out (Above line repeated 20 or so times for different nodes.) Thanks, Kent Hanson

6 9

Slurm version 24.11 is now available
by Tim Wickberg 29 Nov '24

29 Nov '24

We are pleased to announce the availability of the Slurm 24.11 release. To highlight some new features in 24.11: - New gpu/nvidia plugin. This does not rely on any NVIDIA libraries, and will build by default on all systems. It supports basic GPU detection and management, but cannot currently identify GPU-to-GPU links, or provide usage data as these are not exposed by the kernel driver. - Add autodetected GPUs to the output from "slurmd -C". - Added new QOS-based reports to "sreport". - Revamped network I/O with the "conmgr" thread-pool model. - Added new "hostlist function" syntax for management commands and configuration files. - switch/hpe_slingshot - Added support for hardware collectives setup through the fabric manager. (Requires SlurmctldParameters=enable_stepmgr) - Added SchedulerParameters=bf_allow_magnetic_slot configuration option to allow backfill planning for magnetic reservations. - Added new "scontrol listjobs" and "liststeps" commands to complement "listpids", and provide --json/--yaml output for all three subcommands. - Allow jobs to be submitted against multiple QOSes. - Added new experimental "oracle" backfill scheduling support, which permits jobs to be delayed if the oracle function determines the reduced fragmentation of the network topology is sufficiently advantageous. - Improved responsiveness of the controller when jobs are requeued by replacing the "db_index" identifier with a slurmctld-generated unique identifier. ("SLUID") - New options to job_container/tmpfs to permit site-specific scripts to modify the namespace before user steps are launched, and to ensure all steps are completely captured within that namespace. The Slurm documentation has also been updated to the 24.11 release. (Older versions can be found in the archive, linked from the main documentation page.) Slurm can be downloaded from https://www.schedmd.com/download-slurm/ . - Tim -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

Non-intiutive rank placement/CPU masking
by Ohlerich, Martin 28 Nov '24

28 Nov '24

Dear * , I've some question for understanding. Essentially, I use the following job script: ---------------------------> #!/bin/bash #SBATCH -J srun_test #SBATCH --time=0:02:00 #SBATCH --export=NONE #SBATCH --partition=test #SBATCH --nodes=2 #SBATCH -o ./%x.%j.out #SBATCH -D . srun --export=all --mpi=pmi2 --verbose --cpu-bind=verbose -n 96 --ntasks-per-node=48 --ntasks-per-socket=24 -c 2 -m block:block -B 2:56:1 /lrz/sys/tools/placement_test_2021/bin/placement-test.intel_impi ---------------------------> On a sapphire rapid with 2 sockets and 56 physical CPUs. The executable is just some MPI dummy, build with Intel Compilers and Intel MPI (although I guess that's not essential here). The related MPI environment looks as show below (1). Slurm environment is shown below under (2). Now. what I would expect intuitively is that I place here 24 ranks per socket. And here also on the even CPUs in sequence starting from CPU 0 of each socket. What I get is srun: defined options srun: -------------------- -------------------- srun: (null) : i20r01c01s[06,08] srun: cpu-bind : verbose srun: cpus-per-task : 2 srun: distribution : block:block srun: export : all srun: extra-node-info : 2:56:1 srun: jobid : 395726 srun: job-name : MGLET_srun srun: mpi : pmi2 srun: nodes : 2 srun: ntasks : 96 srun: ntasks-per-node : 48 srun: ntasks-per-socket : 24 srun: verbose : 1 srun: -------------------- -------------------- srun: end of defined options srun: jobid 395726: nodes(2):`i20r01c01s[06,08]', cpu counts: 224(x2) srun: Implicitly setting --exact, because -c/--cpus-per-task given. srun: CpuBindType=verbose,threads srun: launching StepId=395726.0 on host i20r01c01s06, 48 tasks: [0-47] srun: launching StepId=395726.0 on host i20r01c01s08, 48 tasks: [48-95] srun: topology/tree: init: topology tree plugin loaded cpu-bind=MASK - i20r01c01s06, task 1 1 [98437]: mask 0xc set cpu-bind=MASK - i20r01c01s06, task 2 2 [98438]: mask 0x30 set cpu-bind=MASK - i20r01c01s06, task 3 3 [98439]: mask 0xc0 set cpu-bind=MASK - i20r01c01s06, task 4 4 [98440]: mask 0x300 set cpu-bind=MASK - i20r01c01s06, task 5 5 [98441]: mask 0xc00 set cpu-bind=MASK - i20r01c01s06, task 6 6 [98442]: mask 0x3000 set cpu-bind=MASK - i20r01c01s06, task 7 7 [98443]: mask 0xc000 set cpu-bind=MASK - i20r01c01s06, task 8 8 [98444]: mask 0x30000 set cpu-bind=MASK - i20r01c01s06, task 9 9 [98445]: mask 0xc0000 set cpu-bind=MASK - i20r01c01s06, task 10 10 [98446]: mask 0x300000 set cpu-bind=MASK - i20r01c01s06, task 11 11 [98447]: mask 0xc00000 set cpu-bind=MASK - i20r01c01s06, task 12 12 [98448]: mask 0x3000000 set cpu-bind=MASK - i20r01c01s06, task 13 13 [98449]: mask 0xc000000 set cpu-bind=MASK - i20r01c01s06, task 14 14 [98450]: mask 0x30000000 set cpu-bind=MASK - i20r01c01s06, task 15 15 [98451]: mask 0xc0000000 set cpu-bind=MASK - i20r01c01s06, task 16 16 [98452]: mask 0x300000000 set cpu-bind=MASK - i20r01c01s06, task 17 17 [98453]: mask 0xc00000000 set cpu-bind=MASK - i20r01c01s06, task 18 18 [98454]: mask 0x3000000000 set cpu-bind=MASK - i20r01c01s06, task 19 19 [98455]: mask 0xc000000000 set cpu-bind=MASK - i20r01c01s06, task 20 20 [98456]: mask 0x30000000000 set cpu-bind=MASK - i20r01c01s06, task 21 21 [98457]: mask 0xc0000000000 set cpu-bind=MASK - i20r01c01s06, task 22 22 [98458]: mask 0x300000000000 set cpu-bind=MASK - i20r01c01s06, task 23 23 [98459]: mask 0xc00000000000 set cpu-bind=MASK - i20r01c01s06, task 24 24 [98460]: mask 0x300000000000000 set cpu-bind=MASK - i20r01c01s06, task 25 25 [98461]: mask 0xc00000000000000 set cpu-bind=MASK - i20r01c01s06, task 26 26 [98462]: mask 0x3000000000000000 set cpu-bind=MASK - i20r01c01s06, task 27 27 [98463]: mask 0xc000000000000000 set cpu-bind=MASK - i20r01c01s06, task 28 28 [98464]: mask 0x30000000000000000 set cpu-bind=MASK - i20r01c01s06, task 29 29 [98465]: mask 0xc0000000000000000 set cpu-bind=MASK - i20r01c01s06, task 30 30 [98466]: mask 0x300000000000000000 set cpu-bind=MASK - i20r01c01s06, task 31 31 [98467]: mask 0xc00000000000000000 set cpu-bind=MASK - i20r01c01s06, task 32 32 [98468]: mask 0x3000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 33 33 [98469]: mask 0xc000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 34 34 [98470]: mask 0x30000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 35 35 [98471]: mask 0xc0000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 36 36 [98472]: mask 0x300000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 37 37 [98473]: mask 0xc00000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 38 38 [98474]: mask 0x3000000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 39 39 [98475]: mask 0xc000000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 40 40 [98476]: mask 0x30000000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 41 41 [98477]: mask 0xc0000000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 42 42 [98478]: mask 0x300000000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 43 43 [98479]: mask 0xc00000000000000000000000 set cpu-bind=MASK - i20r01c01s06, task 44 44 [98480]: mask 0x3 set cpu-bind=MASK - i20r01c01s06, task 45 45 [98481]: mask 0xc set cpu-bind=MASK - i20r01c01s06, task 46 46 [98482]: mask 0x30 set srun: Node i20r01c01s06, 48 tasks started cpu-bind=MASK - i20r01c01s08, task 48 0 [179006]: mask 0x3 set cpu-bind=MASK - i20r01c01s08, task 49 1 [179007]: mask 0xc set cpu-bind=MASK - i20r01c01s08, task 50 2 [179008]: mask 0x30 set cpu-bind=MASK - i20r01c01s08, task 51 3 [179009]: mask 0xc0 set cpu-bind=MASK - i20r01c01s08, task 52 4 [179010]: mask 0x300 set cpu-bind=MASK - i20r01c01s08, task 53 5 [179011]: mask 0xc00 set cpu-bind=MASK - i20r01c01s08, task 54 6 [179012]: mask 0x3000 set cpu-bind=MASK - i20r01c01s08, task 55 7 [179013]: mask 0xc000 set cpu-bind=MASK - i20r01c01s08, task 56 8 [179014]: mask 0x30000 set cpu-bind=MASK - i20r01c01s08, task 57 9 [179015]: mask 0xc0000 set cpu-bind=MASK - i20r01c01s08, task 58 10 [179016]: mask 0x300000 set cpu-bind=MASK - i20r01c01s08, task 59 11 [179017]: mask 0xc00000 set cpu-bind=MASK - i20r01c01s08, task 60 12 [179018]: mask 0x3000000 set cpu-bind=MASK - i20r01c01s08, task 61 13 [179019]: mask 0xc000000 set cpu-bind=MASK - i20r01c01s08, task 62 14 [179020]: mask 0x30000000 set cpu-bind=MASK - i20r01c01s08, task 63 15 [179021]: mask 0xc0000000 set cpu-bind=MASK - i20r01c01s08, task 64 16 [179022]: mask 0x300000000 set cpu-bind=MASK - i20r01c01s08, task 65 17 [179023]: mask 0xc00000000 set cpu-bind=MASK - i20r01c01s08, task 66 18 [179024]: mask 0x3000000000 set cpu-bind=MASK - i20r01c01s08, task 67 19 [179025]: mask 0xc000000000 set cpu-bind=MASK - i20r01c01s08, task 68 20 [179026]: mask 0x30000000000 set cpu-bind=MASK - i20r01c01s08, task 69 21 [179027]: mask 0xc0000000000 set cpu-bind=MASK - i20r01c01s08, task 70 22 [179028]: mask 0x300000000000 set cpu-bind=MASK - i20r01c01s08, task 71 23 [179029]: mask 0xc00000000000 set cpu-bind=MASK - i20r01c01s08, task 72 24 [179030]: mask 0x300000000000000 set cpu-bind=MASK - i20r01c01s08, task 73 25 [179031]: mask 0xc00000000000000 set cpu-bind=MASK - i20r01c01s08, task 74 26 [179032]: mask 0x3000000000000000 set cpu-bind=MASK - i20r01c01s08, task 75 27 [179033]: mask 0xc000000000000000 set cpu-bind=MASK - i20r01c01s08, task 76 28 [179034]: mask 0x30000000000000000 set cpu-bind=MASK - i20r01c01s08, task 77 29 [179035]: mask 0xc0000000000000000 set cpu-bind=MASK - i20r01c01s08, task 78 30 [179036]: mask 0x300000000000000000 set cpu-bind=MASK - i20r01c01s08, task 79 31 [179037]: mask 0xc00000000000000000 set cpu-bind=MASK - i20r01c01s08, task 80 32 [179038]: mask 0x3000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 81 33 [179039]: mask 0xc000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 82 34 [179040]: mask 0x30000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 83 35 [179041]: mask 0xc0000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 84 36 [179042]: mask 0x300000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 85 37 [179043]: mask 0xc00000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 86 38 [179044]: mask 0x3000000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 87 39 [179045]: mask 0xc000000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 88 40 [179046]: mask 0x30000000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 89 41 [179047]: mask 0xc0000000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 90 42 [179048]: mask 0x300000000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 91 43 [179049]: mask 0xc00000000000000000000000 set cpu-bind=MASK - i20r01c01s08, task 92 44 [179050]: mask 0x3 set cpu-bind=MASK - i20r01c01s08, task 93 45 [179051]: mask 0xc set cpu-bind=MASK - i20r01c01s08, task 94 46 [179052]: mask 0x30 set cpu-bind=MASK - i20r01c01s08, task 95 47 [179053]: mask 0xc0 set srun: Node i20r01c01s08, 48 tasks started cpu-bind=MASK - i20r01c01s06, task 47 47 [98483]: mask 0xc0 set In words: the last 4 ranks meant for the second socket of each node are actually placed on the first socket ... So, effectively --ntasks-per-socket is then ignored? I couldn't see an obvious reason for that. Maybe I miss some important point ... or neglect some interference with Intel MPI's runtime environment (there didn't change anything in the masks when unsetting KMP_AFFINITY). I'd like also to mention that explicit placement via cpu_map does the right thing. ---------------------------> #!/bin/bash #SBATCH -J srun_test #SBATCH --time=0:02:00 #SBATCH --export=NONE #SBATCH --partition=test #SBATCH --nodes=2 #SBATCH --ntasks-per-node=48 #SBATCH -o ./%x.%j.out #SBATCH -D . srun --export=all --mpi=pmi2 --cpu-bind=map_cpu:$(seq 0 2 46 | tr '\n' ',')$(seq 56 2 102 | tr '\n' ',') /lrz/sys/tools/placement_test_2021/bin/placement-test.intel_impi ---------------------------> So, a workaround is available for me. But if I could get some illuminating hint on where my intuition failed above, I'd be very grateful. Thank you! Cheers, Martin (1) I_MPI_FILESYSTEM=on I_MPI_HYDRA_BOOTSTRAP=slurm I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher I_MPI_HYDRA_BRANCH_COUNT=128 I_MPI_OFFLOAD=0 I_MPI_OFFLOAD_FAST_MEMCPY_COLL=0 I_MPI_OFFLOAD_RDMA=0 I_MPI_OFI_PROVIDER=psm3 I_MPI_ROOT=/dss/lrzsys/sys/spack/release/24.1.0/opt/x86_64/intel-oneapi-mpi/2021.11.0-gcc-w56vuor/mpi/2021.11 KMP_AFFINITY=granularity=thread,compact,1,0 (2) SLURMD_NODENAME=i20r01c01s06 SLURM_CLUSTER_NAME=sng2 SLURM_CONF=/etc/slurm/slurm.conf SLURM_CPUS_ON_NODE=224 SLURM_ECLIBR=0 SLURM_ECPLUG=1 SLURM_ERLAST=sbatch SLURM_ERSBAC=1 SLURM_GET_USER_ENV=0 SLURM_GTIDS=0 SLURM_JOBID=395726 SLURM_JOB_ACCOUNT=pr28fa SLURM_JOB_CPUS_PER_NODE=224(x2) SLURM_JOB_END_TIME=1732781080 SLURM_JOB_GID=3000114 SLURM_JOB_ID=395726 SLURM_JOB_NAME=MGLET_srun SLURM_JOB_NODELIST=i20r01c01s[06,08] SLURM_JOB_NUM_NODES=2 SLURM_JOB_PARTITION=test SLURM_JOB_QOS=test SLURM_JOB_START_TIME=1732780959 SLURM_JOB_UID=3808660 SLURM_JOB_USER=di49zop SLURM_LOCALID=0 SLURM_NNODES=2 SLURM_NODEID=0 SLURM_NODELIST=i20r01c01s[06,08] SLURM_PRIO_PROCESS=0 SLURM_PROCID=0 SLURM_SCRIPT_CONTEXT=prolog_task SLURM_SETUP_LICENSE=open source - no access restrictions SLURM_SETUP_MAINTAINER_LIST=Bader(a)lrz.de SLURM_SUBMIT_DIR=/dss/dsshome1/00/di49zop/test_mglet SLURM_SUBMIT_HOST=login26 SLURM_TASKS_PER_NODE=224(x2) SLURM_TASK_PID=98399 SLURM_TOPOLOGY_ADDR=leaf.m02r05.i20r01c01s06 SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node

1 0

I'm very confused about priority/fairshare calculation
by Cutts, Tim 27 Nov '24

27 Nov '24

I don't know how many times I've read the docs; I keep thinking I understand it, but something is really wrong with prioritisation on our cluster, and we're struggling to understand why. The setup: 1. We have a group who submit two types of work; production jobs and research jobs. 2. We have two sacctmgr accounts for this; let's call those 'prod' and 'research'. 3. We also have some dedicated hardware that they paid for which can be used only by users associated with the prod account. Desired behaviour: 1. Usage of their dedicated hardware by production jobs should not hugely decrease the fairshare priority for research jobs in other partitions. 2. Usage of shared hardware should decrease their fairshare priority (whether by production or research jobs) 3. Memory should make a relatively small contribution to TRES usage (it's not normally the constrained resource) Our approach: Set TRESBillingWeights for cpu, memory and gres/GPU usage on shared partitions. Typically these are set to: CPU=1.0,Mem=0.25G,GRES/gpu=1.0 Set TRESBillingWeights to something small on the dedicated hardware partition, such as: CPU=0.25 Set PriorityWeightFairshare and PriorityWeightAge to values such that Fairshare dominates when jobs are young, and Age takes over if they've been pending a long time The observed behaviour: 1. production association jobs have a high priority; this is working well 2. research jobs are still getting heavily penalised in fairshare, and we don't understand why; they seem to have enormous RawUsage, largely coming from memory: Here's what I see from sshare (sensitive details removed, obviously): sshare -l -A prod, research -a -o Account,RawUsage,EffectvUsage,FairShare,LevelFS,TRESRunMins%80 | grep -v cpu=0 > ' Account RawUsage EffectvUsage FairShare LevelFS TRESRunMins -------------------- ----------- ------------- ---------- ---------- -------------------------------------------------------------------------------- prod 1587283 0.884373 0.226149 cpu=81371,mem=669457237,energy=0,node=20610,billing=100833,fs/disk=0,vmem=0,pag+ prod 1082008 0.681681 0.963786 0.366740 cpu=81281,mem=669273429,energy=0,node=20520,billing=100833,fs/disk=0,vmem=0,pag+ prod 505090 0.318202 0.964027 0.785664 cpu=90,mem=184320,energy=0,node=90,billing=0,fs/disk=0,vmem=0,pages=0,gres/gpu=+ research 1043560787 0.380577 0.121648 cpu=17181098808,mem=35196566339054,energy=0,node=4295361360,billing=25773481938+ research 146841 0.000141 0.005311 124.679238 cpu=824,mem=3375923,energy=0,node=824,billing=824,fs/disk=0,vmem=0,pages=0,gres+ research 17530141 0.016798 0.001449 1.044377 cpu=254484,mem=3379938816,energy=0,node=161907,billing=893592,fs/disk=0,vmem=0,+ research 167597 0.000161 0.005070 109.238498 cpu=7275,mem=223516160,energy=0,node=7275,billing=50931,fs/disk=0,vmem=0,pages=+ research 12712481 0.012182 0.001931 1.440166 cpu=186327,mem=95399526,energy=0,node=23290,billing=232909,fs/disk=0,vmem=0,pag+ research 11521011 0.011040 0.002173 1.589104 cpu=8167,mem=267626086,energy=0,node=8167,billing=65338,fs/disk=0,vmem=0,pages=+ research 9719735 0.009314 0.002414 1.883599 cpu=15020,mem=69214617,energy=0,node=1877,billing=3755,fs/disk=0,vmem=0,pages=0+ research 25004766 0.023961 0.001207 0.732184 cpu=590778,mem=6464600473,energy=0,node=98910,billing=2266887,fs/disk=0,vmem=0,+ research 68938740 0.066061 0.000724 0.265570 cpu=159332,mem=963064985,energy=0,node=89957,billing=192706,fs/disk=0,vmem=0,pa+ research 7359413 0.007052 0.002656 2.487710 cpu=81401,mem=583487624,energy=0,node=20350,billing=20350,fs/disk=0,vmem=0,page+ research 718714430 0.688714 0.000241 0.025473 cpu=20616,mem=337774728,energy=0,node=5154,billing=92772,fs/disk=0,vmem=0,pages+ research 1016606 0.000974 0.003863 18.009010 cpu=17179774580,mem=35184178340113,energy=0,node=4294943645,billing=25769661870+ Firstly, why are the mem TRES numbers so enormous? Secondly, what's going on with the last user, where the rawusage is tiny, but the TRESRunMins is ridiculously big? That could be messing up the whole thing. Thanks in advance for any advice (either that can help explain what I've misunderstood, or to suggestions of "there's a better way to achieve what you want") Tim -- Tim Cutts Scientific Computing Platform Lead AstraZeneca ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>

1 0

making a maint reservation on a specific GPU
by Paul Raines 22 Nov '24

22 Nov '24

We have a 8 GPU server in which one GPU has gone into an error state that will require a reboot to clear. I have jobs on the server running on good GPUs that will take another 3 days to complete. In the meantime, I would like short jobs to run on the good free GPUs till I reboot. I set a reservation for the time window I plan to reboot on the whole node with scontrol create reservation reservationName=rtx-01_reboot users=root starttime=2024-11-25T06:00:00 duration=720 Nodes=rtx-01 flags=maint,ignore_jobs But I would like to set a reservation on just the bad GPU (gpu_id=7) from now till 2024-11-25T06:00:00 so no job runs that will use it. Is that possible? --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

Suspending jobs and resuming
by Ratnasamy, Fritz 22 Nov '24

22 Nov '24

Hi, I am using an old slurm version 20.11.8 and we had to reboot our cluster today for maintenance. I suspended all the jobs on it with the command scontrol suspend list_job_ids and all the jobs paused and were suspended. However, when I tried to resume them after the reboot, scontrol resume did not work (it was showing in the reason column " (JobHeldAdmin)". I was able to release them with scontrol release and the jobs started to run back. However, the SLURM recorded time on it resetted (Time columns, showing 0:00 for all the jobs) though the jobs seem to have re-started from the last point before he got suspended. 1- Did I follow the right procedure to suspend, reboot and resume/release? 2- In this case, does the wall time for all the jobs goes into reset and therefore anyone with slurm admin rights will be able to have their jobs last longer than the wall time limit by suspending and resuming a job? Best, *Fritz Ratnasamy* Data Scientist Information Technology

2 1

Slurm PID Files
by Matthias Leopold 20 Nov '24

20 Nov '24

Hi, I compiled and installed Slurm 24.05 on Ubuntu 22.04 following this tutorial: https://www.schedmd.com/slurm/installation-tutorial/ Systemd service files are from deb packages that result from this. Do I have to worry that slurmctld and slurmd don't write PID files although SlurmctldPidFile and SlurmdPidFile are defined in slurm.conf? Paths for PID files exist and are writeable, logs don't show any error. slurmdbd does write a PID file as defined in slurmdbd.conf. thx Matthias

1 0

Does Slurm support DSP
by shaobo liu 20 Nov '24

20 Nov '24

Dear all Does slurm support DSP (Digital Signal Processing)? slurm website does not see DSP related content。

3 4

Does Slurm support DSP
by shaobo liu 19 Nov '24

19 Nov '24

Dear all Does slurm support DSP (Digital Signal Processing)? slurm website does not see DSP related content。

1 0

error: Unable to contact slurm controller (connect failure)
by Daniel Rodriguez Lopez (ext) 19 Nov '24

19 Nov '24

Dear all, We recently tried to fix our version of slurm in every node of our cluster. After the instalation (slurm 20.11.9) in one of the compute nodes, most of the commads (squeue, sinfo, scontrol show config etc) returns this error: error: Unable to contact slurm controller (connect failure) The .log files don't show any errors, we have both debugs values equal to debug5. Also, the rest of the cluster works as usual. I appreciate any insight on what could be the cause. Thank you and regards, Daniel

4 3

Slurm services status=1/FAILURE for CentOs cluster
by Mojtaba Farrokhbin 19 Nov '24

19 Nov '24

<https://superuser.com/posts/1862116/timeline> I have reinstalled slurm resource management on a HPC cluster. But it seems there is a problem on starting slurmd services. Here is the system status: "systemctl status slurmd" shows: ● slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sun 2024-11-10 11:31:24 +0130; 1 weeks 1 days ago ------------------------------------------- The slurm.conf output is: # # Example slurm.conf file. Please run configurator.html # (in doc/html) to build a configuration file customized # for your environment. # # # slurm.conf file generated by configurator.html. # # See the slurm.conf man page for more information. # ClusterName=linux ControlMachine=master.cluster.... #ControlAddr= #BackupController= #BackupAddr= # SlurmUser=slurm #SlurmdUser=root SlurmctldPort=6817 SlurmdPort=6818 AuthType=auth/munge #JobCredentialPrivateKey= #JobCredentialPublicCertificate= StateSaveLocation=/var/spool/slurm/ctld SlurmdSpoolDir=/var/spool/slurm/d SwitchType=switch/none MpiDefault=none SlurmctldPidFile=/var/run/slurmctld.pid SlurmdPidFile=/var/run/slurmd.pid #ProctrackType=proctrack/pgid ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup TaskPluginParam=cpusets #PluginDir= #FirstJobId= #MaxJobCount= #PlugStackConfig= #PropagatePrioProcess= #PropagateResourceLimits= #PropagateResourceLimitsExcept= #Prolog= #Epilog= #SrunProlog= #SrunEpilog= #TaskProlog= #TaskEpilog= #TaskPlugin= #TrackWCKey=no #TreeWidth=50 #TmpFS= #UsePAM= # # GPU definition (Added by S 2022/11) GresTypes=gpu # TIMERS SlurmctldTimeout=300 SlurmdTimeout=300 InactiveLimit=0 MinJobAge=300 KillWait=30 Waittime=0 # # SCHEDULING SchedulerType=sched/backfill #SchedulerAuth= #SelectType=select/linear SelectType=select/cons_res # select partial node #SelectTypeParameters=CR_CPU_Memory SelectTypeParameters=CR_Core_Memory #FastSchedule=1 FastSchedule=0 PriorityType=priority/multifactor PriorityDecayHalfLife=0 PriorityUsageResetPeriod=NONE PriorityWeightFairshare=100000 #PriorityWeightAge=1000 PriorityWeightPartition=10000 #PriorityType=priority/multifactor #PriorityDecayHalfLife=14-0 #PriorityUsageResetPeriod=14-0 #PriorityWeightFairshare=100000 #PriorityWeightAge=1000 #PriorityWeightPartition=10000 #PriorityWeightJobSize=1000 #PriorityMaxAge=1-0 DefMemPerCPU=2000 # LOGGING SlurmctldDebug=3 SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=3 SlurmdLogFile=/var/log/slurmd.log JobCompType=jobcomp/none #JobCompLoc= # # ACCOUNTING #JobAcctGatherType=jobacct_gather/linux #JobAcctGatherFrequency=30 # AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=master #AccountingStorageLoc= #AccountingStoragePass= #AccountingStorageUser= # # COMPUTE NODES # OpenHPC default configuration # Edited by Surin 1399/11 #AccountingStorageEnforce=limits AccountingStorageEnforce=QOS,Limits,Associations #TaskPlugin=task/affinity #PropagateResourceLimitsExcept=MEMLOCK #AccountingStorageType=accounting_storage/filetxt Epilog=/etc/slurm/slurm.epilog.clean NodeName=cn0[1-5] NodeHostName=cn0[1-5] RealMemory=128307 Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 Feature=HyperThread State=UNKNOWN NodeName=gp01 NodeHostName=gp01 RealMemory=128307 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 Feature=HyperThread Gres=gpu:4 State=UNKNOWN #PartitionName=all Nodes=cn0[1-5],gp01 MaxTime=INFINITE State=UP Oversubscribe=EXCLUSIVE PartitionName=all Nodes=cn0[1-5],gp01 MaxTime=10-00:00:00 State=UP MaxNodes=1 PartitionName=normal Nodes=cn0[1-5] Default=YES MaxTime=10-00:00:00 State=UP MaxNodes=1 #PartitionName=normal Nodes=cn0[1-5] Default=YES MaxTime=INFINITE State=UP Oversubscribe=EXCLUSIVE PartitionName=gpu Nodes=gp01 MaxTime=10-00:00:00 State=UP SlurmctldParameters=enable_configless ReturnToService=1 HealthCheckProgram=/usr/sbin/nhc HealthCheckInterval=300 ------------------------------------------- scontrol show node cn01 shows: NodeName=cn01 CoresPerSocket=16 CPUAlloc=0 CPUTot=64 CPULoad=N/A AvailableFeatures=HyperThread ActiveFeatures=HyperThread Gres=(null) NodeAddr=cn01 NodeHostName=cn01 RealMemory=128557 AllocMem=0 FreeMem=N/A Sockets=2 Boards=1 State=DOWN* ThreadsPerCore=2 TmpDisk=64278 Weight=1 Owner=N/A MCS_label=N/A Partitions=all,normal BootTime=None SlurmdStartTime=None CfgTRES=cpu=64,mem=128557M,billing=64 AllocTRES= CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s Reason=undraining ------------------------------------------- "scontrol ping" also works as: Slurmctld(primary) at master.cluster... is UP ------------------------------------------- "systemctl start slurmd" shows: Job for slurmd.service failed because the control process exited with error code. See "systemctl status slurmd.service" and "journalctl -xe" for details. ------------------------------------------- "systemctl status slurmd.service" shows: ● slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Mon 2024-11-18 17:02:50 +0130; 41s ago Process: 219025 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=1/FAILURE) Nov 18 17:02:50 master.cluster.... systemd[1]: Starting Slurm node daemon... Nov 18 17:02:50 master.cluster.... slurmd[219025]: fatal: Unable to determine this slurmd's NodeName Nov 18 17:02:50 master.cluster.... systemd[1]: slurmd.service: control process exited, code=exited status=1 Nov 18 17:02:50 master.cluster.... systemd[1]: Failed to start Slurm node daemon. Nov 18 17:02:50 master.cluster.... systemd[1]: Unit slurmd.service entered failed state. Nov 18 17:02:50 master.cluster.... systemd[1]: slurmd.service failed. ------------------------------------------- "journalctl -xe" output is: Nov 18 17:04:54 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:05:04 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:05:08 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:05:12 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:05:22 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:05:26 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:06:34 master.cluster.... munged[2514]: Purged 2 credentials from replay hash ------------------------------------------- slurmd.services contains the following information: [Unit] Description=Slurm node daemon After=munge.service network.target remote-fs.target ConditionPathExists=/etc/slurm/slurm.conf [Service] Type=forking EnvironmentFile=-/etc/sysconfig/slurmd ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID PIDFile=/var/run/slurmd.pid KillMode=process LimitNOFILE=131072 LimitMEMLOCK=infinity LimitSTACK=infinity Delegate=yes [Install] WantedBy=multi-user.target [root@master system]# which slurmd /usr/sbin/slurmd [root@master system]# cat slurmd.service [Unit] Description=Slurm node daemon After=munge.service network.target remote-fs.target ConditionPathExists=/etc/slurm/slurm.conf [Service] Type=forking EnvironmentFile=-/etc/sysconfig/slurmd ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID PIDFile=/var/run/slurmd.pid KillMode=process LimitNOFILE=131072 LimitMEMLOCK=infinity LimitSTACK=infinity Delegate=yes [Install] WantedBy=multi-user.target ------------------------------------------- For "which slurmd" command I get this address: /usr/sbin/slurm ------------------------------------------- ls -l /usr/sbin/slurmd shows: lrwxrwxrwx 1 root root 69 Nov 10 10:54 /usr/sbin/slurmd -> /install/centos7.9/compute_gpu/rootimg/usr/sbin/slurmd

1 0

InvalidAccount
by hmeij＠wesleyan.edu 18 Nov '24

18 Nov '24

Manual compilation of 24.05.4. slurmctld and slurmd run on same server. All works ok but all test jobs end up pending with InvalidAccount message. I do not use slurm database and have not enabled accounting. Can not find an answer for this behavior or a misconfiguration. slurm.conf file was generated using easy config tool. Any ideas how to fix this? Thx, -Henk ## looks like all users have access to test queue [hmeij@sharptail2 slurm]$ sinfo -o "%g %.10R %.20l" GROUPS PARTITION TIMELIMIT all test infinite [hmeij@sharptail2 slurm]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST test* up infinite 1 idle sharptail2 ## simple sleep job [hmeij@sharptail2 slurm]$ sbatch sleep Submitted batch job 8 [hmeij@sharptail2 slurm]$ squeue JOBID PARTITION NAME USER ST TIME NODES CPUS MIN_MEMORY NODELIST(REASON) 8 test sleep hmeij PD 0:00 1 1 1G (InvalidAccount) [hmeij@sharptail2 slurm]$ scontrol show job 8 JobId=8 JobName=sleep UserId=hmeij(8216) GroupId=its(623) MCS_label=N/A Priority=1 Nice=0 Account=(null) QOS=(null) JobState=PENDING Reason=InvalidAccount Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A SubmitTime=2024-11-11T13:27:14 EligibleTime=2024-11-11T13:27:14 AccrueTime=2024-11-11T13:27:14 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-11-11T13:27:14 Scheduler=Main Partition=test AllocNode:Sid=sharptail2:644662 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1-1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:1:1 ReqTRES=cpu=1,mem=1G,node=1,billing=1 AllocTRES=(null) Socks/Node=1 NtasksPerN:B:S:C=1:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=1G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/zfshomes/hmeij/slurm/sleep WorkDir=/zfshomes/hmeij/slurm StdErr=/zfshomes/hmeij/slurm/err StdIn=/dev/null StdOut=/zfshomes/hmeij/slurm/out TresPerTask=cpu=1 ## within a minute or so that InvalidAccount changes to None ( ## but job remains pending; 1-7 stuck over the weekend) [hmeij@sharptail2 slurm]$ squeue JOBID PARTITION NAME USER ST TIME NODES CPUS MIN_MEMORY NODELIST(REASON) 8 test sleep hmeij PD 0:00 1 1 1G (None) ## in the slurmctld.log slurmctld: sched: JobId=8 has invalid account slurmctld: debug: set_job_failed_assoc_qos_ptr: Filling in assoc for JobId=8 Assoc=0 slurmctld: debug: sched: Running job scheduler for full queue. slurmctld: error: _refresh_assoc_mgr_qos_list: no new list given back keeping cached one. ##and the slurm.conf accounting section (both AccountingStorageType lines yield same behavior) #AccountingStorageType= AccountingStorageType=accounting_storage/none #JobAcctGatherFrequency=30 #JobAcctGatherType= ## using SchedulerType = sched/builtin

3 5

Listen to job state changes
by egonle＠posteo.me 18 Nov '24

18 Nov '24

Hello, is there any way to listen to job state changes of slurm 23.x or newer? I’d like to kind of subscribe to job state changes instead of polling for job states. Adding this feature to slurm accounting DB seems to be last option right now, although I’d like to avoid it. Thanks&Best Egonle

4 3

How to power up all ~idle nodes and verify that they have started up without issue programmatically
by Xaver Stiensmeier 15 Nov '24

15 Nov '24

Dear Slurm User list, I would like to startup all ~idle (idle and powered down) nodes and check programmatically if all came up as expected. For context: this is for a program that sets up slurm clusters with on demand cloud scheduling. In the most easiest fashion this could be executing a command like *srun FORALL hostname* which would return the names of the nodes if it succeeds and an error message otherwise. However, there's no such input value like FORALL as far as I am aware. One could use -N{total node number} as all nodes are ~idle when this executes, but I don't know an easy way to get the total number of nodes. Best regards, Xaver

3 5

First setup of slurm with a GPU node
by Patrick Begou 13 Nov '24

13 Nov '24

Hi, I'm using slurm on a small 8 nodes cluster. I've recently added one GPU node with two Nvidia A100, one with 40Gb of RAM and one with 80Gb. As using this GPU resource increase I would like to manage this resource with Gres to avoid usage conflict. But at this time my setup do not works as I can reach a GPU without reserving it: srun -n 1 -p tenibre-gpu ./a.out can use a GPU even if the reservation do not specify this resource (checked with running nvidia-smi on the node). "tenibre-gpu" is a slurm partition with only this gpu node. From the documentation I've created a gres.conf file and it is propagated on all the nodes (9 compute nodes, 1 login node and the management node) and slurmd has been restarted. gres.conf is:* ## GPU setup on tenibre-gpu-0 NodeName=tenibre-gpu-0 Name=gpu Type=A100-40 File=/dev/nvidia0 Flags=nvidia_gpu_env NodeName=tenibre-gpu-0 Name=gpu Type=A100-80 File=/dev/nvidia1 Flags=nvidia_gpu_env * * In slurm.conf I have checked these flags: ## Basic scheduling SelectTypeParameters=CR_Core_Memory SchedulerType=sched/backfill SelectType=select/cons_tres ## Generic resources GresTypes=gpu ## Nodes list .... Nodename=tenibre-gpu-0 RealMemory=257270 Sockets=2 CoresPerSocket=16 ThreadsPerCore=1 State=UNKNOWN .... #partitions PartitionName=tenibre-gpu MaxTime=48:00:00 DefaultTime=12:00:00 DefMemPerCPU=4096 MaxMemPerCPU=8192 Shared=YES State=UP Nodes=tenibre-gpu-0 ... May be I've missed something ? I'm running Slurm 20.11.7-1. Thanks for your advices. Patrick

5 6

Slurm and NVIDIA NVML
by Matthias Leopold 13 Nov '24

13 Nov '24

Hi, I'm trying to compile Slurm with NVIDIA NVML support, but the result is unexpected. I get /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so, but when I do "ldd /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so" there is no reference to /lib/x86_64-linux-gnu/libnvidia-ml.so.1 (which I would expect). ~$ ldd /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so linux-vdso.so.1 (0x00007ffd9a3f4000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0bc2c06000) /lib64/ld-linux-x86-64.so.2 (0x00007f0bc2e47000) /lib/x86_64-linux-gnu/libnvidia-ml.so.1 is present during compilation. Also I can see that NVML headers where found in config.status (else I wouldn't get gpu_nvml.so at all to my understanding). Our old cluster was deployed with NVIDIA deepops (which compiles Slurm on every node) and also has NVML support. There ldd brings the expected result ~$ ldd /usr/local/lib/slurm/gpu_nvml.so ... libnvidia-ml.so.1 => /lib/x86_64-linux-gnu/libnvidia-ml.so.1 (0x00007f3b10120000) ... I can't test actual functionality with my new binaries because I don't have a node with GPUs yet. Am I missing something? thank you Matthias

2 2

Slurm release candidate version 24.11.0rc1 available for testing
by Marshall Garey 12 Nov '24

12 Nov '24

We are pleased to announce the availability of Slurm release candidate 24.11.0rc1. To highlight some new features coming in 24.11: - New gpu/nvidia plugin. This does not rely on any NVIDIA libraries, and will build by default on all systems. It supports basic GPU detection and management, but cannot currently identify GPU-to-GPU links, or provide usage data as these are not exposed by the kernel driver. - Add autodetected GPUs to the output from "slurmd -C". - Added new QOS-based reports to "sreport". - Revamped network I/O with the "conmgr" thread-pool model. - Added new "hostlist function" syntax for management commands and configuration files. - switch/hpe_slingshot - Added support for hardware collectives setup through the fabric manager. (Requires SlurmctldParameters=enable_stepmgr) - Added SchedulerParameters=bf_allow_magnetic_slot configuration option to allow backfill planning for magnetic reservations. - Added new "scontrol listjobs" and "liststeps" commands to complement "listpids", and provide --json/--yaml output for all three subcommands. - Allow jobs to be submitted against multiple QOSes. - Added new experimental "oracle" backfill scheduling support, which permits jobs to be delayed if the oracle function determines the reduced fragmentation of the network topology is sufficiently advantageous. - Improved responsiveness of the controller when jobs are requeued by replacing the "db_index" identifier with a slurmctld-generated unique identifier. ("SLUID") - New options to job_container/tmpfs to permit site-specific scripts to modify the namespace before user steps are launched, and to ensure all steps are completely captured within that namespace. This is the first release candidate of the upcoming 24.11 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 24.11.x version and we will address them before the first production 24.11.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Container Problems with various oci.conf settings
by Jason Pascucci 12 Nov '24

12 Nov '24

Hi, I'm running on Ubuntu 20.04. I've got a clean configuration of slurmctld and slurmd on one node. 1) I've configured oci.conf to the defaults defined by "OCI.CONF EXAMPLE FOR RUNC USING RUN (RECOMMENDED OVER USING CREATE/START):". I have a container that I can run by hand: runc --rootless true run -b /opt/pilot_results/results.20241108-184831/step1 test sh-4.2# exit exit and it returns. However, when I srun --container=/opt/pilot_results/results.20241108-184831/step1 ls it hangs after completing the ls, and I have to double ctrl-c out of it. 2) I tried using the configuration for RUNC with Create/Start and it hangs on start. 3) I tried using the configuration for CRUN using RUN, I can run the container by hand with crun, but srun fails with: srun --container=/opt/pilot_results/results.20241108-194914/step1 bash bind socket to `/run/user/1008//pd-builds-bench-1.jrp.34.0.0/notify`: Address already in use sync socket closed srun: error: pd-builds-bench-1: task 0: Exited with exit code 1 4) I tried using the configuration for CRUN with Create/Start and it errors repeatedly with: slurmstepd: error: _get_container_state: RunTimeQuery failed rc:256 output:error opening file `/run/user/1008//pd-builds-bench-1.jrp.51.0.0/status`: No such file or directory I went through the (open and closed) support tickets and couldn't find anything that reflects any of these errors, and I'm pretty stuck at this point. Any help would be welcome. Thanks, JRP

1 0