- slurm-users - lists.schedmd.com

Slurmd cannot find cgroup plugin
by Webb, Jordan 20 Jan '25

20 Jan '25

Hi all, When trying to start slurmd, it is failing with cgroup issues. Any suggestions on where to troubleshoot this issue? x8000c0s0b0n0:~ # slurmd -V slurm 24.11.0 x8000c0s0b0n0:~ # slurmd -D -vvv slurmd: debug: Log file re-opened slurmd: debug2: hwloc_topology_init slurmd: debug2: hwloc_topology_load slurmd: debug2: hwloc_topology_export_xml slurmd: debug: CPUs:288 Boards:1 Sockets:4 CoresPerSocket:72 ThreadsPerCore:1 slurmd: error: Couldn't find the specified plugin name for cgroup/v2 looking at all files slurmd: error: cannot find cgroup plugin for cgroup/v2 slurmd: error: cannot create cgroup context for cgroup/v2 slurmd: error: Unable to initialize cgroup plugin slurmd: error: slurmd initialization failed x8000c0s0b0n0:~ # mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) x8000c0s0b0n0:~ # grep cgroup /etc/slurm/slurm.conf ProctrackType=proctrack/cgroup TaskPlugin=task/affinity,task/cgroup Thanks! Jordan

1 1

Re: Slurm "showpartitions" tool has been updated
by Ole Holm Nielsen 17 Jan '25

17 Jan '25

Hi Robert, What's your version of Slurm? We run 24.05.5. The showpartitions tools compacts the list of nodenames using "scontrol show hostlistsorted <nodelist>". Can you please check your scontrol manual page if "hostlistsorted" is documented? Does the command work without -N? Thanks, Ole On 17-01-2025 19:54, Robert Kudyba wrote: > Hello Ole, > > If you'd like me to post this on the Git page LMK. Any idea why this > error invalid encode argument with -N? > ./showpartitions -N > Print partitions nodelists > Partition statistics for cluster axon at Fri Jan 17 13:51:43 EST 2025 > Partition #Nodes #CPU_cores Cores_pending Job_Nodes > MaxJobTime Cores Mem/Node NODELIST > Name State Total Idle Total Idle Resorc Other Min Max Day- > hr:mn /node (GB) > nklab up 4 0 352 125 0 0 1 infin > 10-00:00 48 191+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > ctn up 3 1 168 152 0 0 1 infin > 10-00:00 56 128 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > issa up 5 1 384 128 0 0 1 infin > 10-00:00 64 191+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > naplab up 3 1 320 231 0 0 1 infin > 10-00:00 64 515+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > lkumar up 3 2 192 191 0 0 1 infin > 10-00:00 64 515 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > u19moc3 up 1 1 96 96 0 0 1 infin > 10-00:00 96 257 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > abbott up 3 2 192 187 0 0 1 infin > 10-00:00 64 515 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > miller up 3 2 192 190 0 0 1 infin > 10-00:00 64 515 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > burst up 25 10 1896 1300 0 0 1 infin > 5-00:00 48 128+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > zrc up 6 1 376 190 0 0 1 infin > 10-00:00 48 128+invalid encode argument > > On Fri, Jan 17, 2025 at 7:31 AM Ole Holm Nielsen via slurm-users <slurm- > users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com>> wrote: > > Dear Slurm users, > > The "showpartitions" tool prints a Slurm cluster partition status > overview > with 1 line per partition. Color coding helps locating under- or > over-utilized partitions. > > The Slurm "showpartitions" tool has now been updated with a new "-N" > flag > which prints the nodename list for each partition. > > You can download this tool from > https://urldefense.proofpoint.com/v2/url? > u=https-3A__github.com_OleHolmNielsen_Slurm-5Ftools_tree_master_partitions&d=DwICAg&c=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM&r=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY&m=SvjV7A7QvPs0H4AmOiKvni-pI3DeHZmNgrCPf9ije3ftTN6STgomGCGHyuspdUFz&s=e2VRHqxuaDlLYggVv8eW9tkQqfXU332ucgmnqiU3Toc&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_OleHolmNiel…> > > If you have any issues or suggestions, please write to > Ole.H.Nielsen(a)fysik.dtu.dk <mailto:Ole.H.Nielsen@fysik.dtu.dk> or > open an issue on GitHub. > > Best regards, > Ole > > -- > Ole Holm Nielsen > PhD, Senior HPC Officer > Department of Physics, Technical University of Denmark > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > <mailto:slurm-users@lists.schedmd.com> > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > <mailto:slurm-users-leave@lists.schedmd.com> >

2 1

Slurm "showpartitions" tool has been updated
by Ole Holm Nielsen 17 Jan '25

17 Jan '25

Dear Slurm users, The "showpartitions" tool prints a Slurm cluster partition status overview with 1 line per partition. Color coding helps locating under- or over-utilized partitions. The Slurm "showpartitions" tool has now been updated with a new "-N" flag which prints the nodename list for each partition. You can download this tool from https://github.com/OleHolmNielsen/Slurm_tools/tree/master/partitions If you have any issues or suggestions, please write to Ole.H.Nielsen(a)fysik.dtu.dk or open an issue on GitHub. Best regards, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

1 0

job_container/tmpfs and srun.
by Phill Harvey-Smith 17 Jan '25

17 Jan '25

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

2 1

Unable to receive password reminder
by Loris Bennett 14 Jan '25

14 Jan '25

Hi, Over a week ago I sent the message below to the address I found for the list owner, but have not received a response. Does anyone know how to proceed in this case? Cheers, Loris -------------------- Start of forwarded message -------------------- From: Loris Bennett <loris.bennett(a)fu-berlin.de> To: <slurm-users-owner(a)lists.schedmd.com> Subject: Unable to receive password reminder Date: Mon, 6 Jan 2025 08:35:42 +0100 Dear list owner, I have recently switched from reading the list via mail to using the mail to news gateway at news.gmane.io. Therefore I would like change my mailman settings in order to stop delivery of postings via mail. As I have forgotten my list password, I requested a reminder. However I get the reply that no user with the given email address was found in the user database. The addresses I tried were loris.bennett(a)fu-berlin.de loris(a)zedat.fu-berlin.de the former being an alias for the latter. This is the email account which to which emails from the list are sent, so I am somewhat confused as to why the neither of the addresses is recognised. Could you please help me to resolve this issue? Regards Loris Bennett -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin -------------------- End of forwarded message -------------------- -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

2 1

Environment Variables per Queue
by Vitorio Cargnini 14 Jan '25

14 Jan '25

Hello, Can environment variables be set up based on a specific queue? How do I set up the environment variables for a given queue? Like we are deploying SLURM to control our jobs of simulation using VCS, and we don't want to make public the LM_LICENSE environment variable to all users. So we would like to set it on the queue once a job is started on that queue would have that environment variable. Regards, Vitorio

2 1

Discrepancy in TmpDisk values between scontrol and sinfo in Slurm24.05.3
by Gizo Nanava 11 Jan '25

11 Jan '25

There seems to be an issue with the TmpDisk value reporting in Slurm 24.05.3. While the correct value is displayed using the scontrol show nodes command, sinfo appears to report an incorrect value under certain conditions. For example, the TmpDisk parameter for my compute nodes is configured as 21800000 (21.8 TB) in slurm.conf, and this value is shown correctly with > scontrol show nodes ai-n007 ...... State=MIXED ThreadsPerCore=1 TmpDisk=21800000 Weight=1 Owner=N/A MCS_label=N/A .... However, when using the sinfo command with the -O Disk:20 option, the displayed value is 2180000, which is an order of magnitude lower: > sinfo -hO Disk:20 -n ai-n007 2180000 It appears that sinfo might be truncating or misinterpreting values with 8 or more digits. Has anyone encountered a similar issue? Is this a known bug in Slurm 24.05.3? Any suggestions or insights would be greatly appreciated. Best regards, Gizo

2 1

Power saving is getting disabled in Slurm
by Suman Deb 10 Jan '25

10 Jan '25

Hello, We are enabling global power saving for slurm. This is our config. But we are experiencing an issue where Slurm is talking nodes out of power saving mode. We are using config less and dynamic nodes. Slurmctrld logs 2025-01-08T09:54:18.096] Cleared POWER_SAVE flag from nodes dev3-cbf-debug-e2std4-s0-[0-1],dev3-cbf-infra-e2std4-s0-[0-1],dev3-cbf-slurmd-e2std4-[0-4] [2025-01-08T09:54:18.097] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:05:09.683] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:12:00.447] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:17:27.077] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:20:06.735] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:26:16.663] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:26:19.017] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:46:25.078] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T15:58:54.080] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T15:58:56.452] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:08:01.993] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:08:02.157] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:14:26.817] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:14:26.971] debug: power_save module disabled, SuspendTime < 0 We manage the infra through Puppet and I can confirm is SuspendTime=3600. Slurm version:- slurm 24.05.2 # SLURM POWER SAVING FEATURS SuspendProgram=/etc/slurm/suspend_nodes_slurm.par ResumeProgram=/etc/slurm/resume_nodes_slurm.par SuspendTimeout=600 ResumeTimeout=900 ResumeRate=100 SuspendRate=100 SuspendTime=3600 SlurmctldParameters=enable_configless,cloud_dns,idle_on_node_suspend DebugFlags=Power

1 0

Unexpected node got allocation
by Dan Healy 09 Jan '25

09 Jan '25

Hello there and good morning from Baltimore. I have a small cluster with 100 nodes. When the cluster is completely empty of all jobs, the first job gets allocated to node 41. In other clusters, the first job gets allocated to mode 01. If I specify node 01, the allocation works perfectly. I have my partition NodeName set as node[01-99], so having node41 used first is a surprise to me. We also have many other partitions which start with node41, but the partition being used for the allocation starts with node01. Does anyone know what would cause this? Thanks, Daniel Healy

3 3

need help with seff script on ubuntu (slurm 21.08)
by Gérard Henry (AMU) 09 Jan '25

09 Jan '25

Hello all and happy new year, i have installed slurm 21.08 on ubuntu 22 LTS, and database for accounting on a remote machine running same release. I got seff from: https://support.schedmd.com/show_bug.cgi?id=1611 but, there are errors: Use of uninitialized value $ncpus in numeric eq (==) at /usr/bin/seff line 72, <DATA> line 602. Use of uninitialized value $lmem in numeric lt (<) at /usr/bin/seff line 109, <DATA> line 602. Job ID: 14 Cluster: cathena2 Use of uninitialized value $user in concatenation (.) or string at /usr/bin/seff line 126, <DATA> line 602. does anyone have a more recent version? i try to enable debug uncommenting the line: 60c60 < #print Dumper($job); --- > print Dumper($job); but it's strange, because if you run the seff command several times in a row, the dump result is very different, even though the job has been completed. It doesn't change the most important values, but how is it possible not to have the same output? Thanks in advance for help, -- Gérard HENRY Institut Fresnel - UMR 7249 +33 413945457 Aix-Marseille Université - Campus Etoile, BATIMENT FRESNEL, Avenue Escadrille Normandie Niemen, 13013 Marseille Site : https://fresnel.fr/ Afin de respecter l'environnement, merci de n'imprimer cet email que si nécessaire.

4 3

launch failed requeued held
by sportlecon sportlecon 08 Jan '25

08 Jan '25

slurm 24.11 - squeue displays reason "launch failed requeued held"

2 2

formatting node names
by Davide DelVento 07 Jan '25

07 Jan '25

Hi all, I remember seeing on this list a slurm command to change a slurm-friendly list such as gpu[01-02],node[03-04,12-22,27-32,36] into a bash friendly list such as gpu01 gpu02 node03 node04 node12 etc I made a note about it but I can't find my note anymore, nor the relevant message. Can someone please refresh my memory? I'll be more careful with such a note this time, I promise! Thanks and happy new year!

6 11

Change config file location of slurmdbd temporarily
by Sven Schulze 07 Jan '25

07 Jan '25

Hey all, Is there a way to change the location of the slurmdbd.conf temporarily when starting slurmdbd? For all other daemons I can specify "-f", but this doesn't seem to work for slurmdbd, is there a way to edit the build files to achieve this? Kind Regards, Sven

3 4

Permission denied for slurmdbd.conf
by sportlecon＠gmail.com 07 Jan '25

07 Jan '25

ls -ls /usr/local/slurm/etc/slurmdbd.conf 4 -rw------- 1 slurm slurm 497 Dec 28 16:34 /usr/local/slurm/etc/slurmdbd.conf sudo -u slurm /usr/local/slurm/sbin/slurmdbd -Dvvv slurmdbd: error: s_p_parse_file: unable to read "/usr/local/slurm/etc/slurmdbd.conf": Permission denied slurmdbd: fatal: Could not open/read/parse slurmdbd.conf file /usr/local/slurm/etc/slurmdbd.conf

3 2

Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions
by sportlecon sportlecon 07 Jan '25

07 Jan '25

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 26 cpu myscript user1 PD 0:00 4 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions) Anyone can help to fix this?

4 3

All GPUs are Usable if no Gres is Defined
by Jacob Gordon 04 Jan '25

04 Jan '25

Hello, We have a two node GPU cluster with 8 NVidia GPUs. GRES is currently configured and works if a user defines it within their sbtach/interactive job submission (--gres=gpu:3). Users only have access to the GPUs they request. However, when they omit “--gres=gpu:n”, they can use every GPU, which interferes with running jobs that used the gres option. I’m at a loss as to why this is happening. Can someone please look at our configuration to see if anything stands out? SLURM Version = 21.08.5 *Slurm.conf* ClusterName=ommit SlurmctldHost=headnode ProctrackType=proctrack/cgroup ReturnToService=2 SlurmdPidFile=/run/slurmd.pid SlurmdSpoolDir=/var/lib/slurm/slurmd StateSaveLocation=/var/lib/slurm/slurmctld SlurmUser=slurm TaskPlugin=task/cgroup SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory AccountingStorageType=accounting_storage/slurmdbd # AccountingStorageType for other resources # AccountingStorageTRES=gres/gpu #DebugFlags=CPU_Bind,gres JobCompType=jobcomp/none JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log DefMemPerCPU=4000 #NodeName=n01 CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=1000000 NodeName=n01 Gres=gpu:nvidia-l40:8 CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=1000000 NodeName=n02 Gres=gpu:nvidia-l40:8 CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=1000000 #Gres config for GPUs GresTypes=gpu PreemptType=preempt/qos PreemptMode=REQUEUE # reset usage after 1 week PriorityUsageResetPeriod=WEEKLY # The job's age factor reaches 1.0 after waiting in the # queue for 2 weeks. PriorityMaxAge=14-0 # This next group determines the weighting of each of the # components of the Multifactor Job Priority Plugin. # The default value for each of the following is 1. PriorityWeightAge=1000 PriorityWeightFairshare=10000 PriorityWeightJobSize=1000 PriorityWeightPartition=1000 PriorityWeightQOS=1500 # Primary partitions PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP PartitionName=all Nodes=n01,n02 Default=YES MaxTime=01:00:00 DefaultTime=00:30:00 State=UP PartitionName=statds Nodes=n01 Default=NO MaxTime=48:00:00 State=UP Priority=100 State=UP OverSubscribe=FORCE AllowAccounts=statds PartitionName=phil Nodes=n02 Default=NO MaxTime=48:00:00 State=UP Priority=100 State=UP OverSubscribe=FORCE AllowAccounts=phil #Set up condo mode # Condo partitions PartitionName=phil_condo Nodes=n02 Default=NO MaxTime=48:00:00 DefaultTime=00:01:00 State=UP Priority=50 OverSubscribe=FORCE AllowQos=normal PartitionName=statds_condo Nodes=n01 Default=NO MaxTime=48:00:00 DefaultTime=00:01:00 State=UP Priority=50 OverSubscribe=FORCE AllowQos=normal JobSubmitPlugins=lua *Gres.conf* NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia0 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia1 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia2 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia3 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia4 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia5 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia6 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia7 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia0 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia1 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia2 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia3 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia4 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia5 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia6 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia7 *Cgroup.conf* CgroupMountpoint="/sys/fs/cgroup" CgroupAutomount=yes CgroupReleaseAgentDir="/etc/slurm/cgroup" AllowedDevicesFile="/etc/slurm/cgroup_allowed_devices_file.conf" ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes *cgroup_allowed_devices_file.conf* /dev/null /dev/urandom /dev/zero /dev/sda* /dev/cpu/*/* /dev/pts/* /dev/nvidia*

3 2

Resource guarantees
by christopher.furbee＠jhuapl.edu 23 Dec '24

23 Dec '24

Hello, Long time SGE admin, new SLURM admin here. I recently started the transition of all my clusters from SGE to SLURM and everything was great until I hit the "Taco Bell" cluster (fake name). Taco Bell supports 4 projects and under SGE we had a priority system setup using projects to balance the queue. For the life of me I have been unable to replicate this in SLURM. We are looking to configure guaranteed resources based on the project. I had thought we could accomplish this with QOS and accounts but so far we have failed. What we would like to end up with is; When project Gordita is running uncontested 100% of the cluster is available. While Gordita is running, if Crunchwrap submits their jobs we want the scheduler to prioritize those jobs until a 75% Gordita, 25% Crunchwrap balance of jobs is reached. No preempting or priority overriding, just as a Gordita job finishes, if Crunchwrap is less than 25%, start a Crunchwrap job. And then maintain that balance until one of the projects jobs are 100% completed. Any assistance or guidance is greatly appreciated.

1 0

Slurm plugin for custom hardware allocation
by Laura Zharmukhametova 23 Dec '24

23 Dec '24

Hello, Is there an existing Slurm plugin for FPGA allocation? If not, can someone please point me in the right direction for how to approach it. Many thanks

2 1

sending mails wit smail on rocky9
by Marcus Wagner 19 Dec '24

19 Dec '24

Hi all, I have a problem with sending mails on rocky 9 via Slurm. One needs to install s-nail to have "/bin/mail" being available. There are some caveats in smail. In the second part (for the message, when the job began) one need to pipe ( eg echo "") into $MAIL, even in a script with no input, s-nail wants to be interactive. but it suffices to echo an empty text to snail. Nonetheless, I don't get any mail through. it seems the, mailprog for some reason gets killed or errors out for some other reason. While it is perfectionally working if run from the console :/ I all the time get in the slurmctld.log the following: 27212:[2024-12-19T15:54:54.935] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27213:[2024-12-19T15:54:54.945] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27214:[2024-12-19T15:54:54.945] error: MailProg returned error, it's output was '' 27395:[2024-12-19T15:55:55.540] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27396:[2024-12-19T15:55:55.551] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27397:[2024-12-19T15:55:55.551] error: MailProg returned error, it's output was '' 27438:[2024-12-19T15:56:55.981] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27439:[2024-12-19T15:56:55.981] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27440:[2024-12-19T15:56:55.992] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27441:[2024-12-19T15:56:55.992] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27442:[2024-12-19T15:56:55.992] error: MailProg returned error, it's output was '' 27443:[2024-12-19T15:56:55.992] error: MailProg returned error, it's output was '' 27450:[2024-12-19T15:56:58.849] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27451:[2024-12-19T15:56:58.859] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 0 any hints? Best Marcus -- Dipl.-Inf. Marcus Wagner stellv. Gruppenleitung IT Center Gruppe: Server, Storage, HPC Abteilung: Systeme und Betrieb RWTH Aachen University Seffenter Weg 23 52074 Aachen Tel: +49 241 80 24383 wagner(a)itc.rwth-aachen.de www.itc.rwth-aachen.de Social-Media-Kanäle des IT Centers: https://blog.rwth-aachen.de/itc/ https://www.facebook.com/itcenterrwth https://www.linkedin.com/company/itcenterrwth https://twitter.com/ITCenterRWTH https://www.youtube.com/c/ITCenterRWTHAachen

1 0

lua, glib, gtk, kafka not found for slurm 24.11 and Alma 8
by Bernd Melchers 17 Dec '24

17 Dec '24

Dear all, i tried to rpmbuild slurm-24.11.0 for Alma Linux 8. Build failed because some installed Packages are not found by slurms configure script: rdkafka, glib, gtp and lua But all these packages are installed and they are found by slurm-24.05.x: librdkafka-1.6.1-1.el8.x86_64 librdkafka-devel-1.6.1-1.el8.x86_64 lua-5.3.4-12.el8.x86_64 lua-devel-5.3.4-12.el8.x86_64 glib2-2.56.4-165.el8_10.x86_64 glib2-devel-2.56.4-165.el8_10.x86_64 gtk2-2.24.32-5.el8.x86_64 gtk2-devel-2.24.32-5.el8.x86_64 gtk3-3.22.30-12.el8_10.x86_64 gtk3-devel-3.22.30-12.el8_10.x86_64 Mit freundlichen Grüßen Bernd Melchers -- Archiv- und Backup-Service | fab-service(a)zedat.fu-berlin.de Freie Universität Berlin | Tel. +49-30-838-55905

2 3

AllocNode:Sid in scontrol but not sacct?
by Chris Taylor 16 Dec '24

16 Dec '24

Does the accounting database keep this? Maybe I'm missing something but I don't see a way to query for it in sacct. Chris

1 1

job_container/tmpfs, TmpFS and TmpDisk usage
by Paul Musset 16 Dec '24

16 Dec '24

Hello, I have multiple questions about the usage of job_container/tmpfs, and the TmpFS and TmpDisk variables 1) If my job_container.conf files contains: ``` BasePath=/mnt/slurm_tmp Shared=true ``` is it important what I set TmpFS to in slurm.conf ? Should I set it to '/mnt/slurm_tmp' or '/tmp' ? 2) What size should I put in TmpDisk ? the size advertised by df ? 3) Finally, is there any recommended file system for the partition used as the job_container/tmpfs BasePath ? Best regards, Paul Musset Max Planck Institute for Brain Research

1 0

Difference between sreport's cpu time usage and sacct -j usage
by KK 14 Dec '24

14 Dec '24

Hi all, I have observed a significant discrepancy in CPU usage time calculations between sreport and sacct, and I would like to understand the underlying cause. Let me share the specific case I encountered when calculating CPU usage time for user zt23132881r from November 1, 2024, to November 30, 2024. 1. sreport Results (995,171 minutes): -------------------------------------------------------------------------------- *[root@master ~]# sreport Cluster UserUtilizationByAccount user=zt23132881r start=2024-11-01 end=2024-11-30--------------------------------------------------------------------------------Cluster/User/Account Utilization 2024-11-01T00:00:00 - 2024-11-29T23:59:59 (2505600 secs)Usage reported in CPU Minutes-------------------------------------------------------------------------------- Cluster Login Proper Name Account Used Energy--------- --------- --------------- --------------- -------- ---------djhpc-po+ zt231328+ zt23132881r zt+ zt23132881r_ba+ 995171 6294875* 2. sacct Results: # Without truncate (1,019,927 minutes / 61,195,668 seconds) *[root@master ~]# sacct -u zt23132881r -S 2024-11-01 -E 2024-11-30 -o "jobid,partition,account,user,alloccpus,cputimeraw,state" -X |awk 'BEGIN{total=0}{total+=$6}END{print total}'61195668* # With truncate (967,165 minutes / 58,029,908 seconds) *[root@master ~]# sacct -u zt23132881r -S 2024-11-01 -E 2024-11-30 -o "jobid,partition,account,user,alloccpus,cputimeraw,state" -X --truncate |awk 'BEGIN{total=0}{total+=$6}END{print total}'58029908* # No -X *[root@master ~]# sacct -u zt23132881r -S 2024-11-01 -E 2024-11-30 -o "jobid,partition,account,user,alloccpus,cputimeraw,state" |awk 'BEGIN{total=0}{total+=$6}END{print total}'61195668* The results show three different values: - *sreport: 995,171 minutes* - *sacct (without truncate): 1,019,927 minutes* - *sacct (with truncate): 967,165 minutes* I would appreciate if someone could explain: - Which of these results is more accurate? - How does sreport calculate CPU usage time? - Why does the --truncate option in sacct lead to different results? Thank you for your assistance in clarifying these discrepancies. Best regards

2 1

Node configuration unavailable when using --mem-per-gpu , for specific GPU type
by Matthew R. Baney 13 Dec '24

13 Dec '24

Hi all, I'm seeing some odd behavior when using the --mem-per-gpu flag instead of the --mem flag to request memory when also requesting all available CPUs on a node (in this example, all available nodes have 32 CPUs): $ srun --ntasks-per-node=8 --cpus-per-task=4 --gpus-per-node=gtx1080ti:1 --mem-per-gpu=1g --pty bash srun: error: Unable to allocate resources: Requested node configuration is not available $ srun --ntasks-per-node=8 --cpus-per-task=4 --gpus-per-node=gtx1080ti:1 --mem=1g --pty bash srun: job 3479971 queued and waiting for resources srun: job 3479971 has been allocated resources $ The nodes in this partition have a mix of gtx1080ti and rtx2080ti GPUs, but only one type of GPU is in any one node. The same behavior does not occur when requesting a (node with a) rtx2080ti instead. Is there something I'm missing that would cause the --mem-per-gpu flag to not be working in this example? Thanks, Matthew

1 0

Adding gres usage to the accounting database
by Mark Dixon 13 Dec '24

13 Dec '24

Hello! Our jobs can ask for dedicated per-node disk space, e.g. "--gres=tmp:1G", where an ephemeral directory is managed by the site prolog/epilog and usage is capped using an xfs project quota. This works well, although we really need to look at job_container/tmpfs. I note that slurm already periodically polls jobs for their memory usage, which finds its way into the accounting database tres_usage fields. What would be really nice is if we could extend this polling so that we feed our xfs project quota utilisation into the accounting database, too: users get feedback on what their jobs need, and we notice people who ask for a fraction of a node and all of the disk space, but not use it. Slurm is famous for being extendible. Is there some sort of plugin hook someone can point me at to have a go with, or am I missing a simpler method, please? Thanks! Mark

1 0

2025

2024

slurm-users ----- 2025 ----- June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users