July 2024 - slurm-users - lists.schedmd.com

job_container/tmpfs and srun.
by Phill Harvey-Smith 17 Jan '25

17 Jan '25

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

2 1

Why AllowAccounts not work in slurm-23.11.6
by daijiangkuicgo＠gmail.com 30 Oct '24

30 Oct '24

I have set AllowAccounts=sunlabc5hpc,root, but it doesn’t seem to work. User c010637 is not part of the sunlabc5hpc account but is still able to use the sunlabc5hpc partition. I have tried setting EnforcePartLimits to ALL, ANY, and NO, but none of these options resolved the issue. [c010637@sl-login ~]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cpu* up infinite 3 mix sl-c[0035,0042-0043] cpu* up infinite 1 idle sl-c0036 gpu up infinite 3 idle sl-c[0045-0047] sunlabc5hpc up infinite 1 idle sl-c0048 [c010637@sl-login ~]$ scontrol show partition sunlabc5hpc PartitionName=sunlabc5hpc AllowGroups=ALL AllowAccounts=sunlabc5hpc,root AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED Nodes=sl-c0048 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=256 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=256,mem=515000M,node=1,billing=256,gres/gpu=8 [c010637@sl-login ~]$ sacctmgr list assoc format=cluster,user,account%20,qos user=$USER Cluster User Account QOS ---------- ---------- -------------------- -------------------- snowhpc c010637 c010637_bank normal [c010637@sl-login ~]$ sacctmgr list account sunlabc5hpc Account Descr Org ---------- -------------------- -------------------- sunlabc5h+ sunlabc5hpc sunlabc5hpc [c010637@sl-login ~]$ sacctmgr show assoc where Account=sunlabc5hpc format=User,Account User Account ---------- ---------- sunlabc5h+ c010751 sunlabc5h+ snowdai sunlabc5h+

5 9

AllowAccounts partition setting
by Marko Markoc 17 Oct '24

17 Oct '24

Hi All, At one point, when we were running slurm 22, we set up a few new partitions with `AllowAccounts` setting enabled. We limit access to these partitions only to a few accounts. We manually create these accounts and users using the `sacctmgr` command. At that time, everything was working. Only users in these accounts were able to run jobs in that partition. Few months ago we upgraded slurm to 23.02.7 but just a couple of days ago we noticed that anyone could submit jobs to these partitions. In addition to that, you can provide any value to the `--account` directive and slurm will accept and record it in the job accounting database. In our initial testing, you could only provide `--account` value that has been previously created. I'm still looking into this but I wanted to check if there was any change in version 23 that could have impacted this? I don't remember seeing anything in release notes. Thanks, Marko

6 8

Slurmctld process error 'double free or corruption' on RHEL 9 (Rocky Linux)
by William VINCENT 07 Oct '24

07 Oct '24

Hello I am writing to report an issue with the Slurmctld process on our RHEL 9 (Rocky Linux) . Twice in the past 5 days, the Slurmctld process has encountered an error that resulted in the service stopping. The error message displayed was "double free or corruption (out)". This error has caused significant disruption to our jobs, and we are concerned about its recurrence. We have tried troubleshooting the issue, but we have not been able to identify the root cause of the problem. We would appreciate any assistance or guidance you can provide to help us resolve this issue. Please let us know if you need any additional information or if there are any specific steps we should take to diagnose the problem further. Thank you for your attention to this matter. Best regards, _________________________ Jul 09 22:12:01 admin slurmctld[711010]: double free or corruption (fasttop) Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Main process exited, code=killed, status=6/ABRT Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Failed with result 'signal'. Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Consumed 11min 26.451s CPU time. ..... Jul 14 10:15:01 admin slurmctld[1633720]: double free or corruption (out) Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Main process exited, code=killed, status=6/ABRT Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Failed with result 'signal'. Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Consumed 7min 27.596s CPU time. _________________________ slurmctld -V slurm 22.05.9 ________________________ cat /etc/slurm/slurm.conf |grep -v '#' ClusterName=xxx SlurmctldHost=admin SlurmctldParameters=enable_configless SlurmUser=slurm AuthType=auth/munge CryptoType=crypto/munge SlurmctldPort=6817 StateSaveLocation=/var/spool/slurmctld SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmctldDebug=verbose DebugFlags=NO_CONF_HASH SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmdLogFile=/var/log/slurm/slurmd.log SlurmdDebug=verbose SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core,CR_LLN DefMemPerCPU=1024 MaxMemPerCPU=4096 GresTypes=gpu ProctrackType=proctrack/cgroup JobAcctGatherType=jobacct_gather/cgroup JobAcctGatherFrequency=15 JobCompType=jobcomp/none TaskPlugin=task/cgroup LaunchParameters=use_interactive_step AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=admin AccountingStoragePort=6819 AccountingStorageEnforce=associations AccountingStorageTRES=gres/gpu MailProg=/usr/bin/mailx EnforcePartLimits=YES MaxArraySize=200000 MaxJobCount=500000 MpiDefault=none ReturnToService=2 SwitchType=switch/none TmpFS=/tmpslurm/ UsePAM=1 InactiveLimit=0 KillWait=30 MessageTimeout=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 PriorityType=priority/multifactor PriorityFlags=FAIR_TREE,MAX_TRES PriorityDecayHalfLife=1-0 PriorityWeightFairshare=10000 NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=3500 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=2 Sockets=2 RealMemory=1700 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=1700 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=3500 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=r9nc-24-[1-12] NodeHostname=r9nc-24-[1-12] Sockets=2 CoresPerSocket=12 ThreadsPerCore=1 CPUs=24 RealMemory=180000 State=UNKNOWN NodeName=r9nc-48-[1-4] NodeHostname=r9nc-48-[1-4] Sockets=2 CoresPerSocket=24 ThreadsPerCore=1 CPUs=48 RealMemory=480000 State=UNKNOWN NodeName=r9ng-1080-[1-7] NodeHostname=r9ng-1080-[1-7] Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 CPUs=20 RealMemory=180000 State=UNKNOWN Gres=gpu:1080ti:4 NodeName=r9ng-1080-8 NodeHostname=r9ng-1080-8 Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 CPUs=20 RealMemory=176687 State=UNKNOWN Gres=gpu:1080ti:1 PartitionName=24CPUNodes Nodes=r9nc-24-[1-12] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=7500 DefMemPerCPU=7500 TRESBillingWeights="CPU=1.0,Mem=0.125G" Default=YES PartitionName=48CPUNodes Nodes=r9nc-48-[1-4] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=10000 DefMemPerCPU=8000 TRESBillingWeights="CPU=1.0,Mem=0.125G" PartitionName=GPUNodes Nodes=r9ng-1080-[1-7] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=9000 DefMemPerCPU=9000 PartitionName=GPUNodes1080-dev Nodes=r9ng-1080-8 State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=9000 DefMemPerCPU=9000 Hidden=Yes _________________________ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST 24CPUNodes* up infinite 12 idle r9nc-24-[1-12] 48CPUNodes up infinite 2 idle r9nc-48-[1-2] GPUNodes up infinite 4 idle r9ng-1080-[4-7] GPUNodes1080-dev up infinite 1 idle r9ng-1080-8 -- William VINCENT Administrateur systèmes et réseaux

4 15

Job Step State
by Emyr James 01 Oct '24

01 Oct '24

Dear all, I am working on a script to take completed job accounting data from the slurm accounting database and insert the equivalent data into a clickhouse table for fast reporting I can see that all the information is included in the cluster_job_table and cluster_job_step_table which seem to be joined on job_db_inx To get the cpu usage and peak memory usage etc. I can see that I need to parse the tres columns in the job steps. I couldn't find any column called MaxRSS in the database even though the sacct command prints this. I then found some data in tres_table and assume that sacct is using this. Please correct me if I'm wrong and if sacct is getting information from somwhere other than the accounting database? for the state column I get this... select state, count(*) as num from crg_step_table group by state order by num desc limit 10; +-------+--------+ | state | num | +-------+--------+ | 3 | 590635 | | 5 | 28345 | | 4 | 4401 | | 11 | 962 | | 1 | 8 | +-------+--------+ When I use sacct I see statuses seach as COMPLETED, OUT_OF_MEMORY etc. so there must be a mapping somewhere between these state ids and that text. Can someone prvide that mapping or point me to where it's defined in the database or in the code ? Many thanks, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

2 2

Can't schedule on cloud node: State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING
by Xaver Stiensmeier 20 Sep '24

20 Sep '24

Dear slurm-user list, I have a cloud node that is powered up and down on demand. Rarely it can happen that slurm's resumeTimeout is reached and the node is therefore powered down. We have set ReturnToService=2 in order to avoid the node being marked down, because the instance behind that node is created on demand and therefore after a failure nothing stops the system to start the node again as it is a different instance. I thought this would be enough, but apparently the node is still marked with "NOT_RESPONDING" which leads to slurm not trying to schedule on it. After a while NOT_RESPONDING is removed, but I would like to move it directly from within my fail script if possible so that the node can return to service immediately and not be blocked by "NOT_RESPONDING". Best regards, Xaver

2 3

_refresh_assoc_mgr_qos_list: no new list given back keeping cached one
by joao.damas＠syngenta.com 05 Aug '24

05 Aug '24

Hi all, We are doing a simple setup for a Slurm cluster (version 23.11.6). We follow the documentation and we are trying a setup still without accounting or slurmdbd. The slurm.conf is really simple: ``` ClusterName=Develop SlurmctldHost=head # Slurm configuration AuthType=auth/munge CryptoType=crypto/munge SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdLogFile=/var/log/slurm/slurmd.log SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurmctld # Nodes NodeName=worker1 CoresPerSocket=2 Sockets=1 ThreadsPerCore=1 NodeName=worker2 CoresPerSocket=2 Sockets=1 ThreadsPerCore=1 # Partitions PartitionName=develop Default=YES MaxTime=UNLIMITED Nodes="worker1,worker2" ``` When running a simple `srun sleep 10`, all works well and the log file shows: [2024-05-15T12:34:12.741] sched: _slurm_rpc_allocate_resources JobId=1 NodeList=worker1 usec=549 [2024-05-15T12:34:22.775] _job_complete: JobId=1 WEXITSTATUS 0 [2024-05-15T12:34:22.775] _job_complete: JobId=1 done But when creating a scrip with the same sleep command, and submiting using `sbatch test.sh`, the log shows: [2024-05-15T12:35:39.916] _slurm_rpc_submit_batch_job: JobId=2 InitPrio=1 usec=368 [2024-05-15T12:35:40.000] error: _refresh_assoc_mgr_qos_list: no new list given back keeping cached one. [2024-05-15T12:35:40.000] sched: JobId=2 has invalid account [2024-05-15T12:35:40.145] sched/backfill: _start_job: Started JobId=2 in develop on worker1 [2024-05-15T12:35:50.172] _job_complete: JobId=2 WEXITSTATUS 0 [2024-05-15T12:35:50.172] _job_complete: JobId=2 done We have the same account with the UID and GID, as said in the documentation. Looking at the function that seems to spit out that error (https://github.com/SchedMD/slurm/blob/e9f28ede27795f525e62f998cb2d40931d884…), it appears like there should be some accounting setup? We do not have slurmdbd setup and the documentation states we should test basic functionality before implementing that daemon. Any tips? Thanks in advance. João

4 3

problem with squeue --json with version 24.05.1
by Markus Köberl 05 Aug '24

05 Aug '24

$ squeue --version slurm 24.05.1 $ squeue --json malloc(): invalid size (unsorted) Aborted forcing an older data_parser version works: $ squeue --json=v0.0.40 regards Markus Köberl -- Markus Koeberl Graz University of Technology Signal Processing and Speech Communication Laboratory E-mail: markus.koeberl(a)tugraz.at

4 5

slumrestd 24.05.1: crashes when GET on /slurm/v0.0.41/nodes : unsorted double linked list corrupted
by Josef Dvořáček 01 Aug '24

01 Aug '24

Isn't this failure familiar to anyone? When I ask API endpoint "localhost:6820/slurm/v0.0.41/jobs", slurmrestd segrafults with unsorted double linked list corrupted. Anyone using this API endpoint without segfaulting? I do the get using curl: curl --header X-SLURM-USER-NAME:root --header X-SLURM-USER-TOKEN:$SLURM_JWT -G localhost:6820/slurm/v0.0.41/jobs In comparison, curl --header X-SLURM-USER-NAME:root --header X-SLURM-USER-TOKEN:$SLURM_JWT -G localhost:6820/slurm/v0.0.41/nodes Works well. josef čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: _on_url: [[localhost]:52909] url path: /slurm/v0.0.41/jobs query: (null) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: operations_router: [[localhost]:52909] GET /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: slurmrestd: operations_router: [[localhost]:52909] GET /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:52909] attempting user_name root token authentication pass through čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:52909] attempting user_name root token authentication pass through čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi.json"](0, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi.yaml"](1, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi"](2, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=2 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi","v3"](3, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match shares to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed shares čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","shares"](4, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match reconfigure to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed reconfigure čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","reconfigure"](5, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match diag to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed diag čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","diag"](6, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match ping to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed ping čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","ping"](7, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match licenses to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed licenses čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","licenses"](8, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: method skip for ["slurm","v0.0.41","job","submit"](9, GET != POST) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","job","submit"](9, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: method skip for ["slurm","v0.0.41","job","allocate"](10, GET != POST) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","job","allocate"](10, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match jobs to jobs: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match successful for ["slurm","v0.0.41","jobs"](11, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: operations_router: [[localhost]:52909] found callback handler: (0x0) callback_tag=0 path=/slurm/v0.0.41/jobs parser=data_parser/v0.0.41 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] did not provide a known content type header. Assuming URL encoded. čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _parse_http_accept_entry: found */* with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] accepts */* with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] found accepts */*=application/json with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug3: _resolve_mime: [[localhost]:52909] mime read: application/x-www-form-urlencoded write: application/json čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug3: _call_handler: [[localhost]:52909] BEGIN: calling ctxt handler: 0x7F9C9D294A36[0] for path: /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: wrap_openapi_ctxt_callback: [[localhost]:52909] GET using data_parser/v0.0.41 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x1 from 0x408376 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x408376 from 0x1 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: accounting_storage/slurmdbd: _connect_dbd_conn: Sent PersistInit msg čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x1 from 0x408376 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x408376 from 0x1 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: malloc(): unsorted double linked list corrupted čec 24 14:37:55 slurmserver2.koios.lan systemd[1]: Started Process Core Dump (PID 1502951/UID 0).

2 3

Slurm versions 24.05.2, 23.11.9, and 23.02.8 are now available (security fix for switch plugins)
by Tim Wickberg 31 Jul '24

31 Jul '24

Slurm versions 24.05.2, 23.11.9, and 23.02.8 are now available and include a fix for a recently discovered security issue with the switch plugins. SchedMD customers were informed on July 17th and provided a patch on request; this process is documented in our security policy. [1] For the switch/hpe_slingshot and switch/nvidia_imex plugins, a user could override the isolation between Slingshot VNIs or IMEX channels. If you do not have one of these switch plugins configured, then you are not impacted by this issue. It is unclear what, if any, information could be accessed with access to an unauthorized channel. This disclosure is being made out of an abundance of caution. If you do have one of these plugins enabled, the slurmctld must be restarted before the slurmd daemons to avoid disruption. Downloads are available at https://www.schedmd.com/downloads.php . Release notes follow below. - Tim [1] https://www.schedmd.com/security-policy/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support > * Changes in Slurm 24.05.2 > ========================== > -- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when > more than 10 threads try to get energy at the same time. This prevented > the possibility to get energy from slurmd by any step until slurmd was > restarted, so losing energy accounting metrics in the node. > -- accounting_storage/mysql - Fix issue where new user with wckey did not > have a default wckey sent to the slurmctld. > -- slurmrestd - Prevent slurmrestd segfault when handling the following > endpoints when none of the optional parameters are specified: > 'DELETE /slurm/v0.0.40/jobs' > 'DELETE /slurm/v0.0.41/jobs' > 'GET /slurm/v0.0.40/shares' > 'GET /slurm/v0.0.41/shares' > 'GET /slurmdb/v0.0.40/instance' > 'GET /slurmdb/v0.0.41/instance' > 'GET /slurmdb/v0.0.40/instances' > 'GET /slurmdb/v0.0.41/instances' > 'POST /slurm/v0.0.40/job/{job_id}' > 'POST /slurm/v0.0.41/job/{job_id}' > -- Fix IPMI energy gathering when no IPMIPowerSensors are specified in > acct_gather.conf. This situation resulted in an accounted energy of 0 > for job steps. > -- Fix a minor memory leak in slurmctld when updating a job dependency. > -- scontrol,squeue - Fix regression that caused incorrect values for > multisocket nodes at '.jobs[].job_resources.nodes.allocation' for > 'scontrol show jobs --(json|yaml)' and 'squeue --(json|yaml)'. > -- slurmrestd - Fix regression that caused incorrect values for > multisocket nodes at '.jobs[].job_resources.nodes.allocation' to be dumped > with endpoints: > 'GET /slurm/v0.0.41/job/{job_id}' > 'GET /slurm/v0.0.41/jobs' > -- jobcomp/filetxt - Fix truncation of job record lines > 1024 characters. > -- Fixed regression that prevented compilation on FreeBSD hosts. > -- switch/hpe_slingshot - Drain node on failure to delete CXI services. > -- Fix a performance regression from 23.11.0 in cpu frequency handling when no > CpuFreqDef is defined. > -- Fix one-task-per-sharing not working across multiple nodes. > -- Fix inconsistent number of cpus when creating a reservation using the > TRESPerNode option. > -- data_parser/v0.0.40+ - Fix job state parsing which could break filtering. > -- Prevent cpus-per-task to be modified in jobs where a -c value has been > explicitly specified and the requested memory constraints implicitly > increase the number of CPUs to allocate. > -- slurmrestd - Fix regression where args '-s v0.0.39,dbv0.0.39' and > '-d v0.0.39' would result in 'GET /openapi/v3' not registering as a valid > possible query resulting in 404 errors. > -- slurmrestd - Fix memory leak for dbv0.0.39 jobs query which occurred if the > query parameters specified account, association, cluster, constraints, > format, groups, job_name, partition, qos, reason, reservation, state, users, > or wckey. This affects the following endpoints: > 'GET /slurmdb/v0.0.39/jobs' > -- slurmrestd - In the case the slurmdbd does not respond to a persistent > connection init message, prevent the closed fd from being used, and instead > emit an error or warning depending on if the connection was required. > -- Fix 24.05.0 regression that caused the slurmdbd not to send back an error > message if there is an error initializing a persistent connection. > -- Reduce latency of forwarded x11 packets. > -- Add "curr_dependency" (representing the current dependency of the job) > and "orig_dependency" (representing the original requested dependency of > the job) fields to the job record in job_submit.lua (for job update) and > jobcomp.lua. > -- Fix potential segfault of slurmctld configured with > SlurmctldParameters=enable_rpc_queue from happening on reconfigure. > -- Fix potential segfault of slurmctld on its shutdown when rate limitting > is enabled. > -- slurmrestd - Fix missing job environment for SLURM_JOB_NAME, > SLURM_OPEN_MODE, SLURM_JOB_DEPENDENCY, SLURM_PROFILE, SLURM_ACCTG_FREQ, > SLURM_NETWORK and SLURM_CPU_FREQ_REQ to match sbatch. > -- Add missing bash-completions dependency to slurm-smd-client debian package. > -- Fix bash-completions installation in debian pacakges. > -- Fix GRES environment variable indices being incorrect when only using a > subset of all GPUs on a node and the --gres-flags=allow-task-sharing option > -- Add missing mariadb/mysql client package dependency to debian package. > -- Fail the debian package build early if mysql cannot be found. > -- Prevent scontrol from segfaulting when requesting scontrol show reservation > --json or --yaml if there is an error retrieving reservations from the > slurmctld. > -- switch/hpe_slingshot - Fix security issue around managing VNI access. > -- switch/nvidia_imex - Fix security issue managing IMEX channel access. > -- switch/nvidia_imex - Allow for compatibility with job_container/tmpfs. > * Changes in Slurm 23.11.9 > ========================== > -- Fix many commands possibly reporting an "Unexpected Message Received" when > in reality the connection timed out. > -- Fix heterogeneous job components not being signaled with scancel --ctld and > 'DELETE slurm/v0.0.40/jobs' if the job ids are not explicitly given, > the heterogeneous job components match the given filters, and the > heterogeneous job leader does not match the given filters. > -- Fix regression from 23.02 impeding job licenses from being cleared. > -- Move error to log_flag which made _get_joules_task error to be logged to the > user when too many rpcs were queued in slurmd for gathering energy. > -- slurmrestd - Prevent a slurmrestd segfault when modifying an association > without specifying max TRES limits in the request if those TRES > limits are currently defined in the association. This affects the following > fields of endpoint 'POST /slurmdb/v0.0.38/associations/': > 'associations/max/tres/per/job' > 'associations/max/tres/per/node' > 'associations/max/tres/total' > 'associations/max/tres/minutes/per/job' > 'associations/max/tres/minutes/total' > -- Fix power_save operation after recovering from a failed reconfigure. > -- scrun - Delay shutdown until after start requested. This caused scrun > to never start or shutdown and hung forever when using --tty. > -- Fix backup slurmctld potentially not running the agent when taking over as > the primary controller. > -- Fix primary controller not running the agent when a reconfigure of the > slurmctld fails. > -- jobcomp/{elasticsearch,kafka} - Avoid sending fields with invalid date/time. > -- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when > more than 10 threads try to get energy at the same time. This prevented > the possibility to get energy from slurmd by any step until slurmd was > restarted, so losing energy accounting metrics in the node. > -- slurmrestd - Fix memory leak for dbv0.0.39 jobs query which occurred if the > query parameters specified account, association, cluster, constraints, > format, groups, job_name, partition, qos, reason, reservation, state, users, > or wckey. This affects the following endpoints: > 'GET /slurmdb/v0.0.39/jobs' > -- switch/hpe_slingshot - Fix security issue around managing VNI access. > * Changes in Slurm 23.02.8 > ========================== > -- Fix rare deadlock when a dynamic node registers at the same time that a > once per minute background task occurs. > -- Fix assertion in developer mode on a failed message unpack. > -- switch/hpe_slingshot - Fix security issue around managing VNI access.

1 0

2025

2024

slurm-users July 2024

2025

2024

slurm-users July 2024 ----- 2025 ----- February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users July 2024