December 2024 - slurm-users - lists.schedmd.com

job_container/tmpfs and srun.
by Phill Harvey-Smith 17 Jan '25

17 Jan '25

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

2 1

Permission denied for slurmdbd.conf
by sportlecon＠gmail.com 07 Jan '25

07 Jan '25

ls -ls /usr/local/slurm/etc/slurmdbd.conf 4 -rw------- 1 slurm slurm 497 Dec 28 16:34 /usr/local/slurm/etc/slurmdbd.conf sudo -u slurm /usr/local/slurm/sbin/slurmdbd -Dvvv slurmdbd: error: s_p_parse_file: unable to read "/usr/local/slurm/etc/slurmdbd.conf": Permission denied slurmdbd: fatal: Could not open/read/parse slurmdbd.conf file /usr/local/slurm/etc/slurmdbd.conf

3 2

Resource guarantees
by christopher.furbee＠jhuapl.edu 23 Dec '24

23 Dec '24

Hello, Long time SGE admin, new SLURM admin here. I recently started the transition of all my clusters from SGE to SLURM and everything was great until I hit the "Taco Bell" cluster (fake name). Taco Bell supports 4 projects and under SGE we had a priority system setup using projects to balance the queue. For the life of me I have been unable to replicate this in SLURM. We are looking to configure guaranteed resources based on the project. I had thought we could accomplish this with QOS and accounts but so far we have failed. What we would like to end up with is; When project Gordita is running uncontested 100% of the cluster is available. While Gordita is running, if Crunchwrap submits their jobs we want the scheduler to prioritize those jobs until a 75% Gordita, 25% Crunchwrap balance of jobs is reached. No preempting or priority overriding, just as a Gordita job finishes, if Crunchwrap is less than 25%, start a Crunchwrap job. And then maintain that balance until one of the projects jobs are 100% completed. Any assistance or guidance is greatly appreciated.

1 0

Slurm plugin for custom hardware allocation
by Laura Zharmukhametova 23 Dec '24

23 Dec '24

Hello, Is there an existing Slurm plugin for FPGA allocation? If not, can someone please point me in the right direction for how to approach it. Many thanks

2 1

sending mails wit smail on rocky9
by Marcus Wagner 19 Dec '24

19 Dec '24

Hi all, I have a problem with sending mails on rocky 9 via Slurm. One needs to install s-nail to have "/bin/mail" being available. There are some caveats in smail. In the second part (for the message, when the job began) one need to pipe ( eg echo "") into $MAIL, even in a script with no input, s-nail wants to be interactive. but it suffices to echo an empty text to snail. Nonetheless, I don't get any mail through. it seems the, mailprog for some reason gets killed or errors out for some other reason. While it is perfectionally working if run from the console :/ I all the time get in the slurmctld.log the following: 27212:[2024-12-19T15:54:54.935] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27213:[2024-12-19T15:54:54.945] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27214:[2024-12-19T15:54:54.945] error: MailProg returned error, it's output was '' 27395:[2024-12-19T15:55:55.540] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27396:[2024-12-19T15:55:55.551] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27397:[2024-12-19T15:55:55.551] error: MailProg returned error, it's output was '' 27438:[2024-12-19T15:56:55.981] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27439:[2024-12-19T15:56:55.981] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27440:[2024-12-19T15:56:55.992] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27441:[2024-12-19T15:56:55.992] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27442:[2024-12-19T15:56:55.992] error: MailProg returned error, it's output was '' 27443:[2024-12-19T15:56:55.992] error: MailProg returned error, it's output was '' 27450:[2024-12-19T15:56:58.849] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27451:[2024-12-19T15:56:58.859] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 0 any hints? Best Marcus -- Dipl.-Inf. Marcus Wagner stellv. Gruppenleitung IT Center Gruppe: Server, Storage, HPC Abteilung: Systeme und Betrieb RWTH Aachen University Seffenter Weg 23 52074 Aachen Tel: +49 241 80 24383 wagner(a)itc.rwth-aachen.de www.itc.rwth-aachen.de Social-Media-Kanäle des IT Centers: https://blog.rwth-aachen.de/itc/ https://www.facebook.com/itcenterrwth https://www.linkedin.com/company/itcenterrwth https://twitter.com/ITCenterRWTH https://www.youtube.com/c/ITCenterRWTHAachen

1 0

lua, glib, gtk, kafka not found for slurm 24.11 and Alma 8
by Bernd Melchers 17 Dec '24

17 Dec '24

Dear all, i tried to rpmbuild slurm-24.11.0 for Alma Linux 8. Build failed because some installed Packages are not found by slurms configure script: rdkafka, glib, gtp and lua But all these packages are installed and they are found by slurm-24.05.x: librdkafka-1.6.1-1.el8.x86_64 librdkafka-devel-1.6.1-1.el8.x86_64 lua-5.3.4-12.el8.x86_64 lua-devel-5.3.4-12.el8.x86_64 glib2-2.56.4-165.el8_10.x86_64 glib2-devel-2.56.4-165.el8_10.x86_64 gtk2-2.24.32-5.el8.x86_64 gtk2-devel-2.24.32-5.el8.x86_64 gtk3-3.22.30-12.el8_10.x86_64 gtk3-devel-3.22.30-12.el8_10.x86_64 Mit freundlichen Grüßen Bernd Melchers -- Archiv- und Backup-Service | fab-service(a)zedat.fu-berlin.de Freie Universität Berlin | Tel. +49-30-838-55905

2 3

AllocNode:Sid in scontrol but not sacct?
by Chris Taylor 17 Dec '24

17 Dec '24

Does the accounting database keep this? Maybe I'm missing something but I don't see a way to query for it in sacct. Chris

1 1

job_container/tmpfs, TmpFS and TmpDisk usage
by Paul Musset 16 Dec '24

16 Dec '24

Hello, I have multiple questions about the usage of job_container/tmpfs, and the TmpFS and TmpDisk variables 1) If my job_container.conf files contains: ``` BasePath=/mnt/slurm_tmp Shared=true ``` is it important what I set TmpFS to in slurm.conf ? Should I set it to '/mnt/slurm_tmp' or '/tmp' ? 2) What size should I put in TmpDisk ? the size advertised by df ? 3) Finally, is there any recommended file system for the partition used as the job_container/tmpfs BasePath ? Best regards, Paul Musset Max Planck Institute for Brain Research

1 0

Difference between sreport's cpu time usage and sacct -j usage
by KK 14 Dec '24

14 Dec '24

Hi all, I have observed a significant discrepancy in CPU usage time calculations between sreport and sacct, and I would like to understand the underlying cause. Let me share the specific case I encountered when calculating CPU usage time for user zt23132881r from November 1, 2024, to November 30, 2024. 1. sreport Results (995,171 minutes): -------------------------------------------------------------------------------- *[root@master ~]# sreport Cluster UserUtilizationByAccount user=zt23132881r start=2024-11-01 end=2024-11-30--------------------------------------------------------------------------------Cluster/User/Account Utilization 2024-11-01T00:00:00 - 2024-11-29T23:59:59 (2505600 secs)Usage reported in CPU Minutes-------------------------------------------------------------------------------- Cluster Login Proper Name Account Used Energy--------- --------- --------------- --------------- -------- ---------djhpc-po+ zt231328+ zt23132881r zt+ zt23132881r_ba+ 995171 6294875* 2. sacct Results: # Without truncate (1,019,927 minutes / 61,195,668 seconds) *[root@master ~]# sacct -u zt23132881r -S 2024-11-01 -E 2024-11-30 -o "jobid,partition,account,user,alloccpus,cputimeraw,state" -X |awk 'BEGIN{total=0}{total+=$6}END{print total}'61195668* # With truncate (967,165 minutes / 58,029,908 seconds) *[root@master ~]# sacct -u zt23132881r -S 2024-11-01 -E 2024-11-30 -o "jobid,partition,account,user,alloccpus,cputimeraw,state" -X --truncate |awk 'BEGIN{total=0}{total+=$6}END{print total}'58029908* # No -X *[root@master ~]# sacct -u zt23132881r -S 2024-11-01 -E 2024-11-30 -o "jobid,partition,account,user,alloccpus,cputimeraw,state" |awk 'BEGIN{total=0}{total+=$6}END{print total}'61195668* The results show three different values: - *sreport: 995,171 minutes* - *sacct (without truncate): 1,019,927 minutes* - *sacct (with truncate): 967,165 minutes* I would appreciate if someone could explain: - Which of these results is more accurate? - How does sreport calculate CPU usage time? - Why does the --truncate option in sacct lead to different results? Thank you for your assistance in clarifying these discrepancies. Best regards

2 1

Node configuration unavailable when using --mem-per-gpu , for specific GPU type
by Matthew R. Baney 13 Dec '24

13 Dec '24

Hi all, I'm seeing some odd behavior when using the --mem-per-gpu flag instead of the --mem flag to request memory when also requesting all available CPUs on a node (in this example, all available nodes have 32 CPUs): $ srun --ntasks-per-node=8 --cpus-per-task=4 --gpus-per-node=gtx1080ti:1 --mem-per-gpu=1g --pty bash srun: error: Unable to allocate resources: Requested node configuration is not available $ srun --ntasks-per-node=8 --cpus-per-task=4 --gpus-per-node=gtx1080ti:1 --mem=1g --pty bash srun: job 3479971 queued and waiting for resources srun: job 3479971 has been allocated resources $ The nodes in this partition have a mix of gtx1080ti and rtx2080ti GPUs, but only one type of GPU is in any one node. The same behavior does not occur when requesting a (node with a) rtx2080ti instead. Is there something I'm missing that would cause the --mem-per-gpu flag to not be working in this example? Thanks, Matthew

1 0

Adding gres usage to the accounting database
by Mark Dixon 13 Dec '24

13 Dec '24

Hello! Our jobs can ask for dedicated per-node disk space, e.g. "--gres=tmp:1G", where an ephemeral directory is managed by the site prolog/epilog and usage is capped using an xfs project quota. This works well, although we really need to look at job_container/tmpfs. I note that slurm already periodically polls jobs for their memory usage, which finds its way into the accounting database tres_usage fields. What would be really nice is if we could extend this polling so that we feed our xfs project quota utilisation into the accounting database, too: users get feedback on what their jobs need, and we notice people who ask for a fraction of a node and all of the disk space, but not use it. Slurm is famous for being extendible. Is there some sort of plugin hook someone can point me at to have a go with, or am I missing a simpler method, please? Thanks! Mark

1 0

Slurm version 24.05.5 is now available
by Marshall Garey 12 Dec '24

12 Dec '24

We are pleased to announce the availability of Slurm version 24.05.5. This release fixes a few potential crashes, several stepmgr bugs, compatibility for sstat and sattach with newer version steps, and some other minor bugs. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.05.5 > ========================== > -- Fix issue signaling cron jobs resulting in unintended requeues. > -- Fix slurmctld memory leak in implementation of HealthCheckNodeState=CYCLE. > -- job_container/tmpfs - Fix SLURM_CONF env variable not being properly set. > -- sched/backfill - Fix job's time_limit being overwritten by time_min for job > arrays in some situations. > -- RoutePart - fix segfault from incorrect memory allocation when node doesn't > exist in any partition. > -- slurmctld - Fix crash when a job is evaluated for a reservation after > removal of a dynamic node. > -- gpu/nvml - Attempt loading libnvidia-ml.so.1 as a fallback for failure in > loading libnvidia-ml.so. > -- slurmrestd - Fix populating non-required object fields of objects as '{}' in > JSON/YAML instead of 'null' causing compiled OpenAPI clients to reject > the response to 'GET /slurm/v0.0.40/jobs' due to validation failure of > '.jobs[].job_resources'. > -- Fix sstat/sattach protocol errors for steps on higher version slurmd's > (regressions since 20.11.0rc1 and 16.05.1rc1 respectively). > -- slurmd - Avoid a crash when starting slurmd version 24.05 with > SlurmdSpoolDir files that have been upgraded to a newer major version of > Slurm. Log warnings instead. > -- Fix race condition in stepmgr step completion handling. > -- Fix slurmctld segfault with stepmgr and MpiParams when running a job array. > -- Fix requeued jobs keeping their priority until the decay thread happens. > -- slurmctld - Fix crash and possible split brain issue if the > backup controller handles an scontrol reconfigure while in control > before the primary resumes operation. > -- Fix stepmgr not getting dynamic node addrs from the controller > -- stepmgr - avoid "Unexpected missing socket" errors. > -- Fix `scontrol show steps` with dynamic stepmgr > -- Support IPv6 in configless mode

1 0

node3 not working - down
by Steven Jones 10 Dec '24

10 Dec '24

Hi, As suggested, 8><--- Stop their services, start them manually one by one (ctld first), then watch whether they talk to each other, and if they don't, learn what stops them from doing so - then iterate editing the config, "scontrol reconfig", lather, rinse, repeat. 8><--- Error logs, On node3, ======== [2024-12-09T21:43:10.694] slurmd started on Mon, 09 Dec 2024 21:43:10 +0000 [2024-12-09T21:43:10.694] CPUs=20 Boards=1 Sockets=20 Cores=1 Threads=1 Memory=48269 TmpDisk=23308 Uptime=3703 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) [2024-12-09T23:38:56.645] error: Munge decode failed: Rewound credential [2024-12-09T23:38:56.645] auth/munge: _print_cred: ENCODED: Tue Dec 10 23:38:30 2024 [2024-12-09T23:38:56.645] auth/munge: _print_cred: DECODED: Mon Dec 09 23:38:56 2024 [2024-12-09T23:38:56.645] error: Check for out of sync clocks [2024-12-09T23:38:56.645] error: slurm_receive_msg_and_forward: g_slurm_auth_verify: REQUEST_PING has authentication error: Invalid authentication credential [2024-12-09T23:38:56.645] error: slurm_receive_msg_and_forward: Protocol authentication error [2024-12-09T23:38:56.655] error: service_connection: slurm_receive_msg: Protocol authentication error [2024-12-10T01:13:18.454] Slurmd shutdown completing ====== I have checked the time, this is fine munge.key I ran md5sum over it and get the same output. When I reboot the warewulf nodes its a common Container file so 1~2 working but not 3 is just wierd. On the controller, ========= [2024-12-10T01:30:35.108] Node node2 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.109] Node node1 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.109] Node node6 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.109] Node node7 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.110] Node node3 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.110] Node node5 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.110] Node node4 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:36.104] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2 ========== So the controller is RHEL9 and the nodes Rocky8 However 1~2 and 4~7 work OK. When I run, [root@vuwunicoslurmd1 log]# scontrol show node node3 NodeName=node3 Arch=x86_64 CoresPerSocket=1 CPUAlloc=0 CPUEfctv=20 CPUTot=20 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=node3 NodeHostName=node3 Version=20.11.9 OS=Linux 4.18.0-553.30.1.el8_10.x86_64 #1 SMP Tue Nov 26 18:56:25 UTC 2024 RealMemory=48000 AllocMem=0 FreeMem=43378 Sockets=20 Boards=1 State=DOWN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=debug BootTime=2024-12-09T20:41:27 SlurmdStartTime=2024-12-10T01:13:25 LastBusyTime=2024-12-10T01:30:31 CfgTRES=cpu=20,mem=48000M,billing=20 AllocTRES= CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s Reason=Node unexpectedly rebooted [slurm@2024-12-09T20:40:27] [root@vuwunicoslurmd1 log]# I get a reason, its like the node is held down / off, does this need to be / can this be cleared? regards Steven

2 2

Job not starting
by Diego Zuccato 10 Dec '24

10 Dec '24

Hello all. An user reported that a job wasn't starting, so I tried to replicate the request and I get: -8<-- [root@ophfe1 root.old]# scontrol show job 113936 JobId=113936 JobName=test.sh UserId=root(0) GroupId=root(0) MCS_label=N/A Priority=1 Nice=0 Account=root QOS=long JobState=PENDING Reason=Priority Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2024-12-06T13:19:36 EligibleTime=2024-12-06T13:19:36 AccrueTime=2024-12-06T13:19:36 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-12-06T13:21:32 Scheduler=Backfill:* Partition=m3 AllocNode:Sid=ophfe1:855189 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1-1 NumCPUs=96 NumTasks=96 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=96,mem=95000M,node=1,billing=1296 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=95000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/home/root.old/test.sh WorkDir=/home/root.old StdErr=/home/root.old/%N-%J.err StdIn=/dev/null StdOut=/home/root.old/%N-%J.out Power= [root@ophfe1 root.old]# scontrol sho partition m3 PartitionName=m3 AllowGroups=ALL DenyAccounts=formazione AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=mtx20 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=CANCEL State=UP TotalCPUs=192 TotalNodes=1 SelectTypeParameters=CR_SOCKET_MEMORY JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=192,mem=1150000M,node=1,billing=2592 TRESBillingWeights=CPU=13.500,Mem=2.2378G [root@ophfe1 root.old]# scontrol show node mtx20 NodeName=mtx20 Arch=x86_64 CoresPerSocket=24 CPUAlloc=0 CPUEfctv=192 CPUTot=192 CPULoad=0.00 AvailableFeatures=ib,matrix,intel,avx ActiveFeatures=ib,matrix,intel,avx Gres=(null) NodeAddr=mtx20 NodeHostName=mtx20 Version=22.05.6 OS=Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47 UTC 2022 RealMemory=1150000 AllocMem=0 FreeMem=1156606 Sockets=4 Boards=1 MemSpecLimit=2048 State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=8 Owner=N/A MCS_label=N/A Partitions=m3 BootTime=2024-12-06T10:01:42 SlurmdStartTime=2024-12-06T10:02:54 LastBusyTime=2024-12-06T10:51:58 CfgTRES=cpu=192,mem=1150000M,billing=2592 AllocTRES= CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s -8<-- So the node is free, the partition does not impose extra limits (used only for accounting factors) but the job does not start. Any hints? Tks -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786

3 5

TRES cpu vs tasks
by Miriam Olmi 10 Dec '24

10 Dec '24

Hi all, I cannot understand the true difference and definition of "core", "task" and "cpu" within the limits associated to a partition via the TRES variable of a QOS. More precisely I have 2 partitions defined as follows: PartitionName=lprod AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=lprod_part DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=1-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED Nodes=r037c01s[01-12] PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=432 TotalNodes=12 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=432,mem=12M,node=12,billing=432 PartitionName=bprod AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=bprod_part DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=1-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED Nodes=r037c01s[01-12] PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=432 TotalNodes=12 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=432,mem=12M,node=12,billing=432 with the two qos defined as: Name Priority GraceTime PreemptMode UsageFactor MaxJobsPU MaxSubmitPU MaxTRES MinTRES Flags -------------------- ---------- ---------- ----------- ----------- --------- ----------- -------------------- -------------------- -------------------------------------------------- lprod_part 0 00:00:00 cluster 1.000000 100 120 cpu=32,mem=366G cpu=1 DenyOnLimit,PartitionMaxNodes,PartitionMinNodes bprod_part 0 00:00:00 cluster 1.000000 100 120 cpu=64,mem=366G cpu=33 DenyOnLimit,OverPartQOS If I try to submit a job on the lprod partition with the directives: #SBATCH --nodes=12 #SBATCH --ntasks-per-node=1 the job is executed correctly while it is not executed if submitted on the bprod partition due to the error: sbatch: error: QOSMinCpuNotSatisfied sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) I understand that this is related to the limit of cpu associated to the partition via the qos: lprod->[1-32]cpus, bprod->[33-64]cpus but I would like to have a more proper explanation since the options I am using are not referring to "cpu" but to "tasks" and I could not find a proper definition of "cpu" in the framework of the TRES. Many thanks in advance. Best, Miriam

2 2

slurm nodes showing down*
by Steven Jones 09 Dec '24

09 Dec '24

I have just rebuilt all my nodes and I see Only 1 & 2 seem available? While 3~6 are not 3's log, [root@node3 log]# tail slurmd.log [2024-12-08T21:45:51.250] CPU frequency setting not configured for this node [2024-12-08T21:45:51.251] slurmd version 20.11.9 started [2024-12-08T21:45:51.252] slurmd started on Sun, 08 Dec 2024 21:45:51 +0000 [2024-12-08T21:45:51.252] CPUs=20 Boards=1 Sockets=20 Cores=1 Threads=1 Memory=48269 TmpDisk=23324 Uptime=30 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) [root@node3 log]# And 7 doesnt want to talk to the controller. [root@node7 slurm]# sinfo slurm_load_partitions: Zero Bytes were transmitted or received [root@node7 slurm]# These are all rebuilt and 1~3 are identical and 4~7 are identical. 7's log keep saying, [2024-12-08T21:49:17.246] error: Unable to register: Zero Bytes were transmitted or received [2024-12-08T21:49:18.263] error: Unable to register: Zero Bytes were transmitted or received [2024-12-08T21:49:19.278] error: Unable to register: Zero Bytes were transmitted or received [2024-12-08T21:49:20.294] error: Unable to register: Zero Bytes were transmitted or received [2024-12-08T21:49:21.310] error: Unable to register: Zero Bytes were transmitted or received [root@vuwunicoslurmd1 slurm]# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST debug* up infinite 2 idle* node[1-2] debug* up infinite 4 down* node[3-6] [root@vuwunicoslurmd1 slurm]# regards Steven

2 4

error and output files
by michaelmorgan937＠gmail.com 09 Dec '24

09 Dec '24

Hi all, I have a program (for example x.x). When I run it as "x.x -I input -o output", I will get an "output" file, as well as output and error files from slurm. However, all the content of slurm output file is in the "output" file (and there are some extra content), so it is a waste to print the slurm output. Is it possible to set directives so that slurm only print error file? Thank you very much. Michael Morgan

2 1

Why is my job killed when ResumeTimeout is reached instead of it being requeued?
by Xaver Stiensmeier 09 Dec '24

09 Dec '24

Dear Slurm-user list, when a job fails because the node startup fails (cloud scheduling), the job should be re-queued: Resume Timeout Maximum time permitted (in seconds) between when a node resume request is issued and when the node is actually available for use. Nodes which fail to respond in this time frame will be marked DOWN and the jobs scheduled on the node requeued. however, instead of requeuing the job, it is killed. [2024-11-18T10:41:52.003] node bibigrid-worker-wubqboa1z2kkgx0-0 not resumed by ResumeTimeout(1200) - marking down and power_save [2024-11-18T10:41:52.003] Killing JobId=1 on failed node bibigrid-worker-wubqboa1z2kkgx0-0 [2024-11-18T10:41:52.046] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 reason set to: FailedStartup [2024-11-18T10:41:52.046] power down request repeating for node bibigrid-worker-wubqboa1z2kkgx0-0 Our ResumeProgram does not change the state of the underlying workers, I think we should set the nodes to DOWN explicitly if the startup fails given: *ResumeProgram* is unable to restore a node to service with a responding slurmd and an updated BootTime, it should set the node state to DOWN, which will result in a requeue of any job associated with the node - this will happen automatically if the node doesn't register within ResumeTimeout but in any case as we can see in the log the job should be requeued based on it reaching the ResumeTimeout alone. I am unsure why that is not happening. The power down request is sent by the ResumeFailProgram. We have SlurmctldParameters=idle_on_node_suspend activated, but that shouldn't affect Resume, I guess. My Slurm version is slurm 23.11.5 Best regards, Xaver # More context ## Slurmctld from submitting job to failure [2024-11-18T10:21:45.490] sched: _slurm_rpc_allocate_resources JobId=1 NodeList=bibigrid-worker-wubqboa1z2kkgx0-0 usec=1221 [2024-11-18T10:21:45.499] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:21:58.387] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:21:58.387] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:22:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:23:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:23:20.009] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:23:23.003] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:23:23.398] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:23:23.398] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:23:53.398] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:23:53.398] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:24:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:24:21.484] slurmscriptd: error: _run_script: JobId=0 resumeprog exit status 1:0 [2024-11-18T10:25:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:26:02.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:26:02.417] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:26:02.417] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:26:20.007] debug: sched: Running job scheduler for full queue. [2024-11-18T10:26:32.417] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:26:32.417] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:27:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:28:20.003] debug: Updating partition uid access list [2024-11-18T10:28:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:28:20.008] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:29:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:29:22.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:29:22.448] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:29:22.448] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:30:20.007] debug: sched: Running job scheduler for full queue. [2024-11-18T10:31:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:32:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:32:42.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:32:42.478] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:32:42.478] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:33:12.479] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:33:12.479] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:33:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:33:20.010] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:34:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:35:20.007] debug: sched: Running job scheduler for full queue. [2024-11-18T10:36:01.004] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:36:01.504] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:36:01.504] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:36:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:36:31.505] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:36:31.505] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:37:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:38:20.008] debug: Updating partition uid access list [2024-11-18T10:38:20.008] debug: sched: Running job scheduler for full queue. [2024-11-18T10:38:20.017] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:39:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:39:21.003] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:39:21.530] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:39:21.530] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:39:51.531] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:39:51.531] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:40:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:41:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:41:52.003] node bibigrid-worker-wubqboa1z2kkgx0-0 not resumed by ResumeTimeout(1200) - marking down and power_save [2024-11-18T10:41:52.003] Killing JobId=1 on failed node bibigrid-worker-wubqboa1z2kkgx0-0 [2024-11-18T10:41:52.046] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 reason set to: FailedStartup [2024-11-18T10:41:52.046] power down request repeating for node bibigrid-worker-wubqboa1z2kkgx0-0 [2024-11-18T10:41:52.047] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:41:52.549] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:41:52.549] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:41:52.736] _slurm_rpc_complete_job_allocation: JobId=1 error Job/step already completing or completed [2024-11-18T10:41:53.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:41:53.000] debug: sched: Running job scheduler for default depth. [2024-11-18T10:41:53.014] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 reason set to: FailedStartup [2024-11-18T10:41:53.014] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 state set to IDLE

1 1

getting slurm going
by Steven Jones 08 Dec '24

08 Dec '24

What tests can I do to prove that slurm is talking to the nodes pls? regards Steven

3 5

Non-Standard Mail Notification in Job
by Daniel Miliate 05 Dec '24

05 Dec '24

Newbie here, I'm trying to send myself an email notification with a custom body/subject for notification of important events. Is this possible? I've tried a few things in the script below. Thanks in advance and apologies if this is not how this listing works! - Daniel ---- Script ---- .... #SBATCH --mail-user= example(a)example.example #SBATCH --export=ALL #SBATCH --mail-type=ALL # Both of these work which mail which mailx EMAIL="example(a)example.example" # Attempt 1 echo "Test email body" | mail -s "Test Subject" "$EMAIL" # Attempt 2 mail -s "hello" "$EMAIL" <<EOF hello world EOF # Attempt 3 echo "Specific Information" | mail -s "SLURM Job information" $EMAIL ---- End ----

3 2

slurm not running on a warewulf node
by Steven Jones 04 Dec '24

04 Dec '24

Hi, I have set a log creation/location in slurm.conf as, SlurmdLogFile=/var/log/slurm/slurmd.log But it is 0 length. Slurm will not run, what else do I need to do to log why its failing pls? regards Steven

4 6

slurmd on a warwwulf node - not running
by Steven Jones 04 Dec '24

04 Dec '24

Hi, I have munge running on the controller and nodes fine, all tests passed. I have slurmctld running on the controller ok after checking the logs /var/spool/slurmctld was not created which I assume should have happened via the rpm install? Anyway I cant get slurmd to run on the warewulf nodes and cant find any log to check? How to fault find this? regards Steven

3 3

How can I make sure my user have only one job per node (Job array --exclusive=user,)
by Oren 04 Dec '24

04 Dec '24

Hi, I have a cluster of 20-nodes, and I want to run a jobarray on that cluster, but I want each node to get one job per node. When I do the following: #!/bin/bash #SBATCH --job-name=process_images_train # Job name #SBATCH --time=50:00:00 # Time limit hrs:min:sec #SBATCH --tasks=1 #SBATCH --cpus-per-task=4 #SBATCH --mem=50000 #SBATCH --array=0-19# 19 # Job array with 20 jobs (0 to 19) I get 10 jobs in node #1 and 10 jobs in node #20, I want a job in each node. I've tried: #SBATCH --exclusive=user Also #SBATCH --spread-job #SBATCH --distribution=cyclic Nothing changes, node#1 got 10 jobs and node#2 got 10 jobs. Thanks

2 6

multiple conf-server entries for sackd
by Brian Andrus 03 Dec '24

03 Dec '24

Not sure anyone would know, but... If you are running slurm in HA mode (multiple SlurmctldHost entries) is it possible to point sackd to more than one using the --conf-server option? So either specify --conf-server more than once, or have a comma-delimited list of them? The docs are a little light about that. Brian Andrus

1 0

sinfo not listing any partitions
by Kent L. Hanson 02 Dec '24

02 Dec '24

I am doing a new install of slurm 24.05.3 I have all the packages built and installed on headnode and compute node with the same munge.key, slurm.conf, and gres.conf file. I was able to run munge and unmunge commands to test munge successfully. Time is synced with chronyd. I can't seem to find any useful errors in the logs. For some reason when I run sinfo no nodes are listed. I just see the headers for each column. Has anyone seen this or know what a next step of troubleshooting would be? I'm new to this and not sure where to go from here. Thanks for any and all help! The odd output I am seeing [username@headnode ~] sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST (Nothing is output showing status of partition or nodes) Slurm.conf ClusterName=slurmkvasir SlurmctldHost=kadmin2 MpiDefault=none ProctrackType=proctrack/cgroup PrologFlags=contain ReturnToService=2 SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmctldPort=6817 SlurmPidFile=/var/run/slurm/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/cgroup MinJobAge=600 SchedulerType=sched/backfill SelectType=select/cons_tres PriorityType=priority/multifactor AccountingStorageHost=localhost AccountingStoragePass=/var/run/munge/munge.socket.2 AccountingStorageType=accounting_storage/slurmdbd AccountingStorageTRES=gres/gpu,cpu,node JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmLogFile=/var/log/slurm/slurmd.log nodeName=k[001-448] PartitionName=default Nodes=k[001-448] Default=YES MaxTime=INFINITE State=up Slurmctld.log Error: Configured MailProg is invalid Slurmctld version 24.05.3 started on cluster slurmkvasir Accounting_storage/slurmdbd: clusteracct_storage_p_register_ctld: Regisetering slurmctld at port 8617 Error: read_slurm_conf: default partition not set. Revovered state of 448 nodes Down nodes: k[002-448] Recovered information about 0 jobs Revovered state of 0 reservations Read_slurm_conf: backup_controller not specified Select/cons_tres; select_p_reconfigure: select/cons_tres: reconfigure Running as primary controller Slurmd.log Error: Node configuration differs from hardware: CPUS=1:40(hw) Boards=1:1(hw) SocketsPerBoard=1:2(hw) CoresPerSocket=1:20(hw) ThreadsPerCore:1:1(hw) CPU frequency setting not configured for this node Slurmd version 24.05.3started Slurmd started on Wed, 27 Nov 2024 06:51:03 -0700 CPUS=1 Boards=1 Cores=1 Threads=1 Memory=192030 TmpDisk=95201 uptime 166740 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) Error: _forward_thread: failed to k019 (10.142.0.119:6818): Connection timed out (Above line repeated 20 or so times for different nodes.) Thanks, Kent Hanson

6 9

2025

2024

slurm-users December 2024

2025

2024

slurm-users December 2024 ----- 2025 ----- February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users December 2024