January 2025 - slurm-users - lists.schedmd.com

Discrepancy in TmpDisk values between scontrol and sinfo in Slurm24.05.3
by Gizo Nanava 11 Jan '25

11 Jan '25

There seems to be an issue with the TmpDisk value reporting in Slurm 24.05.3. While the correct value is displayed using the scontrol show nodes command, sinfo appears to report an incorrect value under certain conditions. For example, the TmpDisk parameter for my compute nodes is configured as 21800000 (21.8 TB) in slurm.conf, and this value is shown correctly with > scontrol show nodes ai-n007 ...... State=MIXED ThreadsPerCore=1 TmpDisk=21800000 Weight=1 Owner=N/A MCS_label=N/A .... However, when using the sinfo command with the -O Disk:20 option, the displayed value is 2180000, which is an order of magnitude lower: > sinfo -hO Disk:20 -n ai-n007 2180000 It appears that sinfo might be truncating or misinterpreting values with 8 or more digits. Has anyone encountered a similar issue? Is this a known bug in Slurm 24.05.3? Any suggestions or insights would be greatly appreciated. Best regards, Gizo

2 1

Power saving is getting disabled in Slurm
by Suman Deb 10 Jan '25

10 Jan '25

Hello, We are enabling global power saving for slurm. This is our config. But we are experiencing an issue where Slurm is talking nodes out of power saving mode. We are using config less and dynamic nodes. Slurmctrld logs 2025-01-08T09:54:18.096] Cleared POWER_SAVE flag from nodes dev3-cbf-debug-e2std4-s0-[0-1],dev3-cbf-infra-e2std4-s0-[0-1],dev3-cbf-slurmd-e2std4-[0-4] [2025-01-08T09:54:18.097] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:05:09.683] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:12:00.447] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:17:27.077] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:20:06.735] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:26:16.663] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:26:19.017] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:46:25.078] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T15:58:54.080] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T15:58:56.452] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:08:01.993] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:08:02.157] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:14:26.817] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:14:26.971] debug: power_save module disabled, SuspendTime < 0 We manage the infra through Puppet and I can confirm is SuspendTime=3600. Slurm version:- slurm 24.05.2 # SLURM POWER SAVING FEATURS SuspendProgram=/etc/slurm/suspend_nodes_slurm.par ResumeProgram=/etc/slurm/resume_nodes_slurm.par SuspendTimeout=600 ResumeTimeout=900 ResumeRate=100 SuspendRate=100 SuspendTime=3600 SlurmctldParameters=enable_configless,cloud_dns,idle_on_node_suspend DebugFlags=Power

1 0

Unexpected node got allocation
by Dan Healy 09 Jan '25

09 Jan '25

Hello there and good morning from Baltimore. I have a small cluster with 100 nodes. When the cluster is completely empty of all jobs, the first job gets allocated to node 41. In other clusters, the first job gets allocated to mode 01. If I specify node 01, the allocation works perfectly. I have my partition NodeName set as node[01-99], so having node41 used first is a surprise to me. We also have many other partitions which start with node41, but the partition being used for the allocation starts with node01. Does anyone know what would cause this? Thanks, Daniel Healy

3 3

need help with seff script on ubuntu (slurm 21.08)
by Gérard Henry (AMU) 09 Jan '25

09 Jan '25

Hello all and happy new year, i have installed slurm 21.08 on ubuntu 22 LTS, and database for accounting on a remote machine running same release. I got seff from: https://support.schedmd.com/show_bug.cgi?id=1611 but, there are errors: Use of uninitialized value $ncpus in numeric eq (==) at /usr/bin/seff line 72, <DATA> line 602. Use of uninitialized value $lmem in numeric lt (<) at /usr/bin/seff line 109, <DATA> line 602. Job ID: 14 Cluster: cathena2 Use of uninitialized value $user in concatenation (.) or string at /usr/bin/seff line 126, <DATA> line 602. does anyone have a more recent version? i try to enable debug uncommenting the line: 60c60 < #print Dumper($job); --- > print Dumper($job); but it's strange, because if you run the seff command several times in a row, the dump result is very different, even though the job has been completed. It doesn't change the most important values, but how is it possible not to have the same output? Thanks in advance for help, -- Gérard HENRY Institut Fresnel - UMR 7249 +33 413945457 Aix-Marseille Université - Campus Etoile, BATIMENT FRESNEL, Avenue Escadrille Normandie Niemen, 13013 Marseille Site : https://fresnel.fr/ Afin de respecter l'environnement, merci de n'imprimer cet email que si nécessaire.

4 3

launch failed requeued held
by sportlecon sportlecon 08 Jan '25

08 Jan '25

slurm 24.11 - squeue displays reason "launch failed requeued held"

2 2

formatting node names
by Davide DelVento 07 Jan '25

07 Jan '25

Hi all, I remember seeing on this list a slurm command to change a slurm-friendly list such as gpu[01-02],node[03-04,12-22,27-32,36] into a bash friendly list such as gpu01 gpu02 node03 node04 node12 etc I made a note about it but I can't find my note anymore, nor the relevant message. Can someone please refresh my memory? I'll be more careful with such a note this time, I promise! Thanks and happy new year!

6 11

Change config file location of slurmdbd temporarily
by Sven Schulze 07 Jan '25

07 Jan '25

Hey all, Is there a way to change the location of the slurmdbd.conf temporarily when starting slurmdbd? For all other daemons I can specify "-f", but this doesn't seem to work for slurmdbd, is there a way to edit the build files to achieve this? Kind Regards, Sven

3 4

Permission denied for slurmdbd.conf
by sportlecon＠gmail.com 07 Jan '25

07 Jan '25

ls -ls /usr/local/slurm/etc/slurmdbd.conf 4 -rw------- 1 slurm slurm 497 Dec 28 16:34 /usr/local/slurm/etc/slurmdbd.conf sudo -u slurm /usr/local/slurm/sbin/slurmdbd -Dvvv slurmdbd: error: s_p_parse_file: unable to read "/usr/local/slurm/etc/slurmdbd.conf": Permission denied slurmdbd: fatal: Could not open/read/parse slurmdbd.conf file /usr/local/slurm/etc/slurmdbd.conf

3 2

Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions
by sportlecon sportlecon 07 Jan '25

07 Jan '25

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 26 cpu myscript user1 PD 0:00 4 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions) Anyone can help to fix this?

4 3

All GPUs are Usable if no Gres is Defined
by Jacob Gordon 05 Jan '25

05 Jan '25

Hello, We have a two node GPU cluster with 8 NVidia GPUs. GRES is currently configured and works if a user defines it within their sbtach/interactive job submission (--gres=gpu:3). Users only have access to the GPUs they request. However, when they omit “--gres=gpu:n”, they can use every GPU, which interferes with running jobs that used the gres option. I’m at a loss as to why this is happening. Can someone please look at our configuration to see if anything stands out? SLURM Version = 21.08.5 *Slurm.conf* ClusterName=ommit SlurmctldHost=headnode ProctrackType=proctrack/cgroup ReturnToService=2 SlurmdPidFile=/run/slurmd.pid SlurmdSpoolDir=/var/lib/slurm/slurmd StateSaveLocation=/var/lib/slurm/slurmctld SlurmUser=slurm TaskPlugin=task/cgroup SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory AccountingStorageType=accounting_storage/slurmdbd # AccountingStorageType for other resources # AccountingStorageTRES=gres/gpu #DebugFlags=CPU_Bind,gres JobCompType=jobcomp/none JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log DefMemPerCPU=4000 #NodeName=n01 CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=1000000 NodeName=n01 Gres=gpu:nvidia-l40:8 CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=1000000 NodeName=n02 Gres=gpu:nvidia-l40:8 CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=1000000 #Gres config for GPUs GresTypes=gpu PreemptType=preempt/qos PreemptMode=REQUEUE # reset usage after 1 week PriorityUsageResetPeriod=WEEKLY # The job's age factor reaches 1.0 after waiting in the # queue for 2 weeks. PriorityMaxAge=14-0 # This next group determines the weighting of each of the # components of the Multifactor Job Priority Plugin. # The default value for each of the following is 1. PriorityWeightAge=1000 PriorityWeightFairshare=10000 PriorityWeightJobSize=1000 PriorityWeightPartition=1000 PriorityWeightQOS=1500 # Primary partitions PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP PartitionName=all Nodes=n01,n02 Default=YES MaxTime=01:00:00 DefaultTime=00:30:00 State=UP PartitionName=statds Nodes=n01 Default=NO MaxTime=48:00:00 State=UP Priority=100 State=UP OverSubscribe=FORCE AllowAccounts=statds PartitionName=phil Nodes=n02 Default=NO MaxTime=48:00:00 State=UP Priority=100 State=UP OverSubscribe=FORCE AllowAccounts=phil #Set up condo mode # Condo partitions PartitionName=phil_condo Nodes=n02 Default=NO MaxTime=48:00:00 DefaultTime=00:01:00 State=UP Priority=50 OverSubscribe=FORCE AllowQos=normal PartitionName=statds_condo Nodes=n01 Default=NO MaxTime=48:00:00 DefaultTime=00:01:00 State=UP Priority=50 OverSubscribe=FORCE AllowQos=normal JobSubmitPlugins=lua *Gres.conf* NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia0 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia1 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia2 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia3 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia4 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia5 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia6 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia7 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia0 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia1 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia2 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia3 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia4 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia5 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia6 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia7 *Cgroup.conf* CgroupMountpoint="/sys/fs/cgroup" CgroupAutomount=yes CgroupReleaseAgentDir="/etc/slurm/cgroup" AllowedDevicesFile="/etc/slurm/cgroup_allowed_devices_file.conf" ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes *cgroup_allowed_devices_file.conf* /dev/null /dev/urandom /dev/zero /dev/sda* /dev/cpu/*/* /dev/pts/* /dev/nvidia*

3 2

2025

2024

slurm-users January 2025

2025

2024

slurm-users January 2025 ----- 2025 ----- February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users January 2025