- slurm-users - lists.schedmd.com

Problem querying slurm batch script archive
by Davide DelVento 09 Jun '25

09 Jun '25

I configured slurm.conf on our machine with AccountingStoreFlags=job_script,job_comment So that people can run sacct -Bj <job_ID> and check what the batch script was for the specified job_ID. This works fine for root and works great for many users (who can see their own job scripts, but not those of other users, as it ought to be). Unfortunately, for several users (but not even close to half of the users of the machine), the sacct -Bj command returns nothing, like if they queried someone else's job, even when querying their own. Does anybody have any clue on what might be wrong and triggering this odd behavior? Thanks!

1 0

seff command not found
by Ratnasamy, Fritz 09 Jun '25

09 Jun '25

Hi, We installed a new slurm version and it returns "command not found" for seff. I do not remember doing any manual installation for the previous versions, I thought it was coming with sacct, sbatch, ect... Any idea how I would need to set it up? I read online seff is actually a perl script. Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Re: Integration testing of Slinky images
by Nicolas Bock 06 Jun '25

06 Jun '25

Hi Brian, sorry for breaking the email thread, my subscription settings were not set correctly and I didn't receive your response as email. Thanks for the tips! I'll give that a try. Best, Nick > Nick, > Presuming you have followed the SchedMD instructions, you should be able > to get a session in your login container: > kubectl --namespace=slurm exec -it statefulsets/slurm-controller -- bash > --login > From there, you can do any standard testing you like. Simple 'srun > hostname' should work to let you know slurm itself is doing it's part. > You can also do commands such as 'scontrol show nodes' to see what and > how many resources you have configured. > Outside that, you need to ensure you have configured your slurm > containers to request the resources you plan on using (eg: gpus) and > that you have enough of them for the script(s) you wish to run. > Brian Andrus

1 0

X11 Forwarding Crashes X Session for too large windows
by Felix Winterhalter 06 Jun '25

06 Jun '25

Hello everyone, we have run into a peculiar issue with our setup. First and foremost let me say that X forwarding using ssh -X or ssh -Y works fine and doesn't show any irregularities. Trying to use the X forwarding of slurm however results in a weird situation. We are using it like: srun --pty --x11 bash And then run something like glxgears. The program starts up normal however whenever you increase the window size beyond a certain size the connection to the X server gets broken and glxgears crashes. The same is true for any other program using X, some don't even start as their inital window size is too large, others crash as soon as you resize them beyond a certain point. We have no clue what might be the cause only that it seems to be related to how slurm does the XForwarding. We've tried X11Parameters both with and without host_authority. We're running 25.05.0 Any help or idea would be very appreciated. Cheers, Felix This message and any attachments are for the sole use of the intended recipient(s). It may contain information that is confidential and privileged. If you are not the intended recipient of this message, you are prohibited from printing, copying, forwarding or saving it. Please delete the message and attachments and notify the sender immediately.

1 1

Integration testing of Slinky images
by Nicolas Bock 06 Jun '25

06 Jun '25

Hi, I was wondering whether there is any integration testing done on the containers and the operator published under the slinky project. Or asked differently, once I have deployed slinky, what tests should I run to confirm that the cluster is working as intended? Thanks! Nick

2 1

X11 performance terrible using plugin
by Simon Andrews 06 Jun '25

06 Jun '25

On our cluster we've noticed that if we use the native x11 slurm plugin (PrologFlags=x11) then X applications work, but are really slow and unresponsive. Even opening menus on graphical application is painfully slow. On the same system if I do a direct ssh connection with ssh -YC from the head node the same applications are quick and responsive. Any suggestions for what might be causing this, and how I can get the native x11 to have the same responsiveness as a direct ssh connection? Many thanks Simon.

6 8

Transfer from GPFS via slurm
by Ratnasamy, Fritz 06 Jun '25

06 Jun '25

Hi, We were told by our hardware provider that large datasets copied from NFS location to GPFS could be conducted via slurm to monitor the transfer. I am not sure of this works as I could not find much online. Otherwise apart from globus and rsync, what would you suggest as a tool/command to copy files from on a cluster between NFS and GPFS? Best, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

Changing account names in sacctmgr
by Burian, John 05 Jun '25

05 Jun '25

All, For reasons I won’t go into, we find we need to change the POSIX group names for all of our users. My understanding is that in the absence of an --account option, sbatch/salloc assumes the account is the user’s primary POSIX group. So, to change group names, we’ll need to change the account names on all the associations in sacctmgr. And ideally, we would change account names in accounting records so we don’t lose accounting history. One is tempted to get into the Slurm DB and start running SQL updates. Is that path too fraught with peril to consider? Or should I just recreate the associations with the new names and take my lumps on the accounting history? John

4 4

error building debug packages on ubuntu 24.04
by Charles Hedrick 03 Jun '25

03 Jun '25

It was unable to find lib/security/pam_slurm_adopt.so It was built as lib64/security, not lib/security. ./debian/slurm-smd-libpam-slurm-adopt.install needs to be lib64/security/pam_slurm_adopt.so More likely, it needs to be built dynamically, so it will work everywhere.

1 0

Slurm version 25.05 is now available
by Tim Wickberg 29 May '25

29 May '25

We are pleased to announce the availability of Slurm 25.05. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md The Slurm documentation has also been updated to the 25.05 release: https://slurm.schedmd.com Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

Slurm accounts managed via ansible
by Ratnasamy, Fritz 28 May '25

28 May '25

Hi, I was wondering whether there might be built in support for managing slurm accounts,users,associations in ansible. it would be nice to be able to organize accounts in a yaml style file and modify accounts settings via gitlab CI/CD. For example, in a format as such: accounts: - name: "Boss1" parent: "root" description: "Big Boos" fairshare: 20 - name: "UnderBoss1" parent: "root" description: "working for Boss1" fairshare: 1 users: -username: "user1" -username: "user2" *Fritz Ratnasamy* Data Scientist Information Technology

3 2

Account getting duplicated
by Ratnasamy, Fritz 28 May '25

28 May '25

Hi, the slurm db duplicates all our account associations. one with cluster=cluster and another where cluster=venus (which is our actual cluster). is that intended? Or should I make any changes? *Fritz Ratnasamy*Data Scientist Information Technology

2 1

Slurm Reporting Difference between sreport and sacct
by Passant Hafez 23 May '25

23 May '25

Hi all, I was wondering if someone can help explaining this discrepancy. I have different values for project gpu consumption using sreport vs sacct (+ some calculations) This is an example that shows this: sreport -t hours -T gres/gpu cluster AccountUtilizationByuser start=2025-04-01 end=2025-04-05 | grep project1234 gives 178 while sacct -n -X --allusers --accounts=project1234 --start=2025-04-01 --end=2025-04-05 -o elapsedraw,AllocTRES%80,user,partition gives 213480 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus 249507 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus 13908 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus 9552 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus 4 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu 11 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu ... I will not bore you with the full output and its calculation, but the first job alone consumed 213480 seconds/60/60 * 8 gpus that's 474.4 gpu hours which is way more than the 178 hrs reported by sreport Any clue why these are inconsistent? or how sreport calculated the 178 value? All the best, Passant

3 3

Crash in "slurmd -C" when latest NVIDIA drivers are used
by Taras Shapovalov 23 May '25

23 May '25

Hello, Does someone have idea why "slurmd -C" crashes when it unloads gpu_nrt.so with latest NVIDIA drivers (570 and 575)? We checked, there is no crash in cuda at the moment and gpu_nvml.so works fine, all nvml calls finish successfully, dlclose on gpu_nvml.so works fine. The crash does not depend whether real GPUs present or not. Steps to reproduce: 1. Install Ubuntu 24.04 2. wget https://download.schedmd.com/slurm/slurm-24.11.4.tar.bz2<https://download.schedmd.com/slurm/slurm-24.11.4.tar.bz2;> 3. tar fx ./slurm-24.11.4.tar.bz2 4. cd slurm-24.11.4 5. apt-get install cuda-12-8 hwloc libmunge-dev -y 6. ./configure 7. make && make install 8. Run "slurmd -C", or sometimes "slurmd -vvv -C" to get the crash. Stack trace: #0 0x0000155555544b2a strlen (ld-linux-x86-64.so.2 + 0x28b2a) #1 0x000015555551fc08 __GI__dl_exception_create (ld-linux-x86-64.so.2 + 0x3c08) #2 0x000015555551d298 __GI__dl_signal_error (ld-linux-x86-64.so.2 + 0x1298) #3 0x000015555551e81d _dl_close (ld-linux-x86-64.so.2 + 0x281d) #4 0x000015555551d51c __GI__dl_catch_exception (ld-linux-x86-64.so.2 + 0x151c) #5 0x000015555551d669 _dl_catch_error (ld-linux-x86-64.so.2 + 0x1669) #6 0x0000155554e97c73 _dlerror_run (libc.so.6 + 0x97c73) #7 0x0000155554e979a6 __dlclose (libc.so.6 + 0x979a6) #8 0x0000155555388a25 gpu_plugin_fini (libslurmfull.so + 0x188a25) #9 0x000015555538f2ef gres_get_autodetected_gpus (libslurmfull.so + 0x18f2ef) #10 0x0000555555564828 _print_config (slurmd + 0x10828) #11 0x0000155554e2a1ca __libc_start_call_main (libc.so.6 + 0x2a1ca) #12 0x0000155554e2a28b __libc_start_main_impl (libc.so.6 + 0x2a28b) #13 0x000055555555fc75 _start (slurmd + 0xbc75) I don't really think the problem is in gpu_nrt itself, seems the problem is in memory corruption somewhere else, but I am not sure. The issue is reproduced constantly. A Best regards, Taras

2 3

Slurm 24.11.x + pyxis + TaskProlog -> fail
by Stefan Staeglich 21 May '25

21 May '25

Hi, we are currently evaluating Slurm 24.11. We are using pyxis in our cluster and have encountered the following problem in connection with TaskProlog: https://github.com/NVIDIA/pyxis/issues/163 Do you know more about the problem and the cause? Best, Stefan

1 0

sacctmgr: error: Sending PersistInit msg: Connection refused
by Ratnasamy, Fritz 18 May '25

18 May '25

Hi, We are working on a test cluster with slurm 24.11.3 and I am getting this error message from the login or compute nodes (note that the error does not show when run from the controller node): sacctmgr list associations tree format=cluster,account,user,maxnodes sacctmgr: error: _open_persist_conn: failed to open persistent connection to host:localhost:6819: Connection refused sacctmgr: error: Sending PersistInit msg: Connection refused Does someone know how fix that issue? Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

New version of slurm
by Ratnasamy, Fritz 16 May '25

16 May '25

Hi, I am trying to install the new version of slurm. Do you know if there is a way to find out what support is compiled into the executables? For example, apache has httpd -L which shows all the loaded modules. See below result: [image: image.png] *Fritz Ratnasamy*Data Scientist Information Technology

3 2

slurmdbd - error: We can't find assoc 1 on cluster slurm
by Edmundo Carmona Antoranz 14 May '25

14 May '25

Hello! We are hitting this problem when we are making a first run of slurmdbd: [2025-05-13T13:29:08.916] debug4: accounting_storage/as_mysql: _get_parent_id: 10(as_mysql_assoc.c:699) query select id_assoc, lineage from "slurm_assoc_table" where user='' and deleted!=1 and acct='root'; [2025-05-13T13:29:08.916] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=(null)(4294967294), acct=(null), cluster=slurm, partition=(null) [2025-05-13T13:29:08.916] debug2: _find_assoc_rec_id: no associations added yet [2025-05-13T13:29:08.916] error: We can't find assoc 1 on cluster slurm I checked and up to that moment the table slurm_assoc_table has not getting a record inserted after the table has been created. [root@ea-r9-29504-pid1 ~]# grep -C1 slurm_assoc_table /var/log/slurmdbd [2025-05-13T13:29:06.666] debug2: DBD_ADD_CLUSTERS: called in CONN 10 [2025-05-13T13:29:06.719] debug4: Table "slurm_assoc_table" doesn't exist, adding [2025-05-13T13:29:06.720] debug2: query alter table "slurm_assoc_table" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default '', modify `acct` tinytext not null, modify `partition` tinytext not null default '', modify `parent_acct` tinytext not null default '', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default '', modify `max_tres_pn` text not null default '', modify `max_tres_mins_pj` text not null default '', modify `max_tres_run_mins` text not null default '', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default '', modify `grp_tres_mins` text not null default '', modify `grp_tres_run_mins` text not null default '', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default '', modify `delta_qos` blob not null default '', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42)); [2025-05-13T13:29:06.729] debug3: query insert into table_defs_table (creation_time, mod_time, table_name, definition) values (1747135746, 1747135746, '"slurm_assoc_table"', 'alter table \"slurm_assoc_table\" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default \'\', modify `acct` tinytext not null, modify `partition` tinytext not null default \'\', modify `parent_acct` tinytext not null default \'\', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default \'\', modify `max_tres_pn` text not null default \'\', modify `max_tres_mins_pj` text not null default \'\', modify `max_tres_run_mins` text not null default \'\', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default \'\', modify `grp_tres_mins` text not null default \'\', modify `grp_tres_run_mins` text not null default \'\', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default \'\', modify `delta_qos` blob not null default \'\', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42));') on duplicate key update definition='alter table \"slurm_assoc_table\" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default \'\', modify `acct` tinytext not null, modify `partition` tinytext not null default \'\', modify `parent_acct` tinytext not null default \'\', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default \'\', modify `max_tres_pn` text not null default \'\', modify `max_tres_mins_pj` text not null default \'\', modify `max_tres_run_mins` text not null default \'\', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default \'\', modify `grp_tres_mins` text not null default \'\', modify `grp_tres_run_mins` text not null default \'\', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default \'\', modify `delta_qos` blob not null default \'\', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42));', mod_time=1747135746; [2025-05-13T13:29:06.775] debug4: Table "slurm_assoc_usage_day_table" doesn't exist, adding -- [2025-05-13T13:29:08.916] debug4: accounting_storage/as_mysql: _get_parent_id: 10(as_mysql_assoc.c:699) query select id_assoc, lineage from "slurm_assoc_table" where user='' and deleted!=1 and acct='root'; [2025-05-13T13:29:08.916] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=(null)(4294967294), acct=(null), cluster=slurm, partition=(null) -- [2025-05-13T13:29:08.921] debug4: accounting_storage/as_mysql: _set_assoc_lft_rgt: 10(as_mysql_assoc.c:737) query select lft, rgt from "slurm_assoc_table" where id_assoc=2; [2025-05-13T13:29:08.922] debug4: accounting_storage/as_mysql: _make_sure_user_has_default_internal: 10(as_mysql_assoc.c:401) query select distinct is_def, acct from "slurm_assoc_table" where user='root' and deleted!=1 ORDER BY is_def desc, creation_time desc LIMIT 1; [2025-05-13T13:29:08.924] debug4: accounting_storage/as_mysql: as_mysql_add_assocs: 10(as_mysql_assoc.c:3999) query Even though the message shows up in the logs as an _error_, the service is able to come up fine, by the way, but is there an easy way to avoid the error message in the first place? BR!

1 0

Slurm release candidate version 25.05.0rc1 is available for testing
by Marshall Garey 13 May '25

13 May '25

We are pleased to announce the availability of Slurm release candidate 25.05.0rc1. To highlight some new features coming in 25.05: - Support for defining multiple topology configurations, and varying them by partition. - Support for tracking and allocating hierarchical resources. - Dynamic nodes can be dynamically added to the topology. - topology/block - Allow for gaps in the block layout. - Support for encrypting all network communication with TLS. - jobcomp/kafka - Optionally send job info at job start as well as job end. - Support an OR operator in --license requests. - switch/hpe_slingshot - Support for > 252 ranks per node. - switch/hpe_slingshot - Support mTLS authentication to the fabric manager. - sacctmgr - Add support for dumping and loading QOSes. - srun - Add new --wait-for-children option to keep the step running until all launched processes have been launched (cgroup/v2 only). - slurmrestd - Add new endpoint for creating reservations. This is the first release candidate of the upcoming 25.05 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 25.05.x version and we will address them before the first production 25.05.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/download-slurm/. The changelog for 25.05.0rc1 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-25.05.md#chang… -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Performance Issues after Update to 24.11.5
by Tilman Hoffbauer 13 May '25

13 May '25

Hello, we are running a SLURM-managed cluster with one control node (g-vm03) and 26 worker nodes (ouga[03-28]) on Rocky 8. We recently updated from 20.11.9 through 23.02.8 to 24.11.0 and then 24.11.5. Since then, we are experiencing performance issues - squeue and scontrol ping are slow to react and sometimes deliver "timeout on send/recv" messages, even with only very few parallel requests. We did not experience these issues with SLURM 20.11.9 before, we did not check the intermediate version 23.02.8 in detail before. In the log of slurmctld, we can also find messages like slurmctld: error: slurm_send_node_msg: [socket:[1272743]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error We thus implemented all recommendations from the high throughput documentation, and did achieve improvements with it (most notably by increasing the maximum number of open files and increasing MessageTimeout and TCPTimeout). For debugging, I attached the slurm.conf, the sdiag output (the server thread count is almost always 1 and sometimes increases to 2), the slurmctld log and the slurmdbd log from a time of high load. We would be very thankful for any input on how restore the old performance. Kind Regards, Tilman Hoffbauer

2 4

Do I have to hold back RAM for worker nodes?
by Xaver Stiensmeier 13 May '25

13 May '25

Dear Slurm-User List, currently, in our slurm.conf, we are setting: SelectType: select/cons_tres SelectTypeParameters: CR_Core and in our node configuration /RealMemory /was basically reduced by an amount to make sure the node always had enough RAM to run the OS. However, this is apparently now how it is supposed to be done: Lowering RealMemory with the goal of setting aside some amount for the OS and not available for job allocations will not work as intended if Memory is not set as a consumable resource in *SelectTypeParameters*. So one of the *_Memory options need to be enabled for that goal to be accomplished. (https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory) This leads to four questions regarding holding back RAM for worker nodes. Answers/help with any of those questions would be appreciated. *1.* Is reserving enough RAM for the worker node's OS and slurmd actually a thing you have to manage? *2.* If so how can we reserve enough RAM for the worker node's OS and slurmd when using CR_Core? *3.* Is that maybe a strong argument against using CR_Core that we overlooked? And semi-related: https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory talks about taking a value in megabytes. *4.* Is RealMemory really expecting megabytes or is it mebibytes? Best regards, Xaver

6 8

slurm failing to start as it doesnt know its node name??????
by Steven Jones 12 May '25

12 May '25

Hi, I am not sure what i have missed but I am getting this error on a compute node. ====== root@vm01no16 ~]# systemctl status slurmd.service × slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Sun 2025-05-11 23:01:29 UTC; 15min ago Process: 1178 ExecStart=/usr/sbin/slurmd --systemd $SLURMD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 1178 (code=exited, status=1/FAILURE) CPU: 3ms May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: Starting Slurm node daemon... May 11 23:01:29 vm01no16.ods.vuw.ac.nz slurmd[1178]: slurmd: fatal: Unable to determine this slurmd's NodeName May 11 23:01:29 vm01no16.ods.vuw.ac.nz slurmd[1178]: fatal: Unable to determine this slurmd's NodeName May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: slurmd.service: Main process exited, code=exited, status=1/FAIL> May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: slurmd.service: Failed with result 'exit-code'. May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: Failed to start Slurm node daemon. [root@vm01no16 ~]# uname -a Linux vm01no16.ods.vuw.ac.nz 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 30 17:38:54 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux [root@vm01no16 ~]# vi /etc/hosts [root@vm01no16 ~]# cat /etc/hostname vm01no16.ods.vuw.ac.nz [root@vm01no16 ~]# tail /etc/hosts # Entry for int01no9.ods.vuw.ac.nz 130.195.87.29 int01no9.ods.vuw.ac.nz int01no9.ods.vuw.ac.nz-default # Entry for vm01no13.ods.vuw.ac.nz 130.195.87.41 vm01no13.ods.vuw.ac.nz vm01no13.ods.vuw.ac.nz-default # Entry for vm01no14.ods.vuw.ac.nz 130.195.87.42 vm01no14.ods.vuw.ac.nz vm01no14.ods.vuw.ac.nz-default # Entry for vm01no15.ods.vuw.ac.nz 130.195.87.43 vm01no15.ods.vuw.ac.nz vm01no15.ods.vuw.ac.nz-default # Entry for vm01no16.ods.vuw.ac.nz 130.195.87.44 vm01no16.ods.vuw.ac.nz vm01no16.ods.vuw.ac.nz-default [root@vm01no16 ~]# ====== regards Steven Jones B.Eng (Hons) Technical Specialist - Linux RHCE Victoria University, Digital Solutions, Level 8 Rankin Brown Building, Wellington, NZ 6012 0064 4 463 6272

2 1

Slurm versions 24.11.5, 24.05.8, and 23.11.11 are now available (CVE-2025-43904)
by Tim Wickberg 07 May '25

07 May '25

Slurm versions 24.11.5, 24.05.8, and 23.11.11 are now available and include a fix for a recently discovered security issue. SchedMD customers were informed on April 23rd and provided a patch on request; this process is documented in our security policy. [1] A mistake with permission handling for Coordinators within Slurm's accounting system can allow a Coordinator to promote a user to Administrator. (CVE-2025-43904) Thank you to Sekou Diakite (HPE) for reporting this. Downloads are available at https://www.schedmd.com/downloads.php . Release notes follow below. - Tim [1] https://www.schedmd.com/security-policy/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support > * Changes in Slurm 24.11.5 > ========================== > -- Return error to scontrol reboot on bad nodelists. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.40 > endpoints. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.41 > endpoints. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.42 > endpoints. > -- data_parser/v0.0.42 - Added +inline_enums flag which modifies the > output when generating OpenAPI specification. It causes enum arrays to not > be defined in their own schema with references ($ref) to them. Instead they > will be dumped inline. > -- Fix binding error with tres-bind map/mask on partial node allocations. > -- Fix stepmgr enabled steps being able to request features. > -- Reject step creation if requested feature is not available in job. > -- slurmd - Restrict listening for new incoming RPC requests further into > startup. > -- slurmd - Avoid auth/slurm related hangs of CLI commands during startup > and shutdown. > -- slurmctld - Restrict processing new incoming RPC requests further into > startup. Stop processing requests sooner during shutdown. > -- slurmcltd - Avoid auth/slurm related hangs of CLI commands during > startup and shutdown. > -- slurmctld: Avoid race condition during shutdown or reconfigure that > could result in a crash due delayed processing of a connection while > plugins are unloaded. > -- Fix small memleak when getting the job list from the database. > -- Fix incorrect printing of % escape characters when printing stdio > fields for jobs. > -- Fix padding parsing when printing stdio fields for jobs. > -- Fix printing %A array job id when expanding patterns. > -- Fix reservations causing jobs to be held for Bad Constraints > -- switch/hpe_slingshot - Prevent potential segfault on failed curl > request to the fabric manager. > -- Fix printing incorrect array job id when expanding stdio file names. > The %A will now be substituted by the correct value. > -- Fix printing incorrect array job id when expanding stdio file names. > The %A will now be substituted by the correct value. > -- switch/hpe_slingshot - Fix vni range not updating on slurmctld restart > or reconfigre. > -- Fix steps not being created when using certain combinations of -c and > -n inferior to the jobs requested resources, when using stepmgr and nodes > are configured with CPUs == Sockets*CoresPerSocket. > -- Permit configuring the number of retry attempts to destroy CXI service > via the new destroy_retries SwitchParameter. > -- Do not reset memory.high and memory.swap.max in slurmd startup or > reconfigure as we are never really touching this in slurmd. > -- Fix reconfigure failure of slurmd when it has been started manually and > the CoreSpecLimits have been removed from slurm.conf. > -- Set or reset CoreSpec limits when slurmd is reconfigured and it was > started with systemd. > -- switch/hpe-slingshot - Make sure the slurmctld can free step VNIs after > the controller restarts or reconfigures while the job is running. > -- Fix backup slurmctld failure on 2nd takeover. > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904. > * Changes in Slurm 24.05.8 > ========================== > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904. > * Changes in Slurm 23.11.11 > =========================== > -- Fixed a job requeuing issue that merged job entries into the same SLUID > when all nodes in a job failed simultaneously. > -- Add ABORT_ON_FATAL environment variable to capture a backtrace from any > fatal() message. > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904.

1 0

Lots of RPC calls and REQUEST_GETPW calls
by Guillette, Jeremy 07 May '25

07 May '25

Hello, I’m trying to figure out why we’ve been seeing an increase in network traffic in our AWS-based cluster, which uses Amazon’s parallel cluster tool. After an incident a couple weeks ago, I turned on debug2 logging on the slurmd processes, and I’m seeing huge numbers of `REQUEST_GETPW` and `REQUEST_GETGR` requests going to the slurmd processes. I briefly turned on debug2 logging for `slurmctld` as well, and I’m seeing lots of RPC calls, but not as many as the `REQUEST_GETPW` requests that I’ve seen on compute node slurmd processes. Here’s a sample from the slurmctld log: ``` [2025-04-28T15:11:05.436] debug2: _slurm_rpc_dump_partitions, size=1253 usec=20 [2025-04-28T15:11:05.450] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2971 [2025-04-28T15:11:05.451] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2971 [2025-04-28T15:11:05.451] debug2: _slurm_rpc_dump_partitions, size=1253 usec=16 [2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2788 [2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2788 [2025-04-28T15:11:05.461] debug2: _slurm_rpc_dump_partitions, size=1253 usec=12 [2025-04-28T15:11:05.517] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2916 [2025-04-28T15:11:05.518] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2916 [2025-04-28T15:11:05.518] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.628] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3405 [2025-04-28T15:11:05.629] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3405 [2025-04-28T15:11:05.629] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2189 [2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2189 [2025-04-28T15:11:05.740] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:05.845] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2209 [2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2209 [2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=4106 [2025-04-28T15:11:05.846] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.847] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=4106 [2025-04-28T15:11:05.847] debug2: _slurm_rpc_dump_partitions, size=1253 usec=11 [2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3400 [2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3400 [2025-04-28T15:11:05.938] debug2: _slurm_rpc_dump_partitions, size=1253 usec=18 [2025-04-28T15:11:06.903] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3449 [2025-04-28T15:11:06.904] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3449 [2025-04-28T15:11:06.904] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.175] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3722 [2025-04-28T15:11:07.176] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3722 [2025-04-28T15:11:07.177] debug2: _slurm_rpc_dump_partitions, size=1253 usec=254 [2025-04-28T15:11:07.205] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=4040 [2025-04-28T15:11:07.206] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=4040 [2025-04-28T15:11:07.206] debug2: _slurm_rpc_dump_partitions, size=1253 usec=17 [2025-04-28T15:11:07.237] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2990 [2025-04-28T15:11:07.238] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2990 [2025-04-28T15:11:07.239] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.284] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2920 [2025-04-28T15:11:07.285] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2920 [2025-04-28T15:11:07.285] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.370] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3236 [2025-04-28T15:11:07.371] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3236 [2025-04-28T15:11:07.371] debug2: _slurm_rpc_dump_partitions, size=1253 usec=17 [2025-04-28T15:11:08.463] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2848 [2025-04-28T15:11:08.464] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2848 [2025-04-28T15:11:08.464] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:08.691] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2627 [2025-04-28T15:11:08.692] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2627 [2025-04-28T15:11:08.692] debug2: _slurm_rpc_dump_partitions, size=1253 usec=18 [2025-04-28T15:11:08.873] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3729 [2025-04-28T15:11:08.874] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3729 [2025-04-28T15:11:08.875] debug2: _slurm_rpc_dump_partitions, size=1253 usec=196 [2025-04-28T15:11:08.881] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3461 [2025-04-28T15:11:08.882] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3461 [2025-04-28T15:11:08.882] debug2: _slurm_rpc_dump_partitions, size=1253 usec=10 ``` And from slurmd: ``` [2025-04-27T19:45:01.353] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.475] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.491] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.501] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.504] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.507] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.513] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.606] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.607] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:04.988] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.992] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.995] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.999] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.011] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.016] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.033] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.045] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.048] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.057] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.073] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.077] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.110] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.144] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.170] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.172] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.174] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.203] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.204] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.207] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.316] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.318] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.321] [59253.batch] debug: Handling REQUEST_GETPW ``` This level of debugging makes the logs pretty huge, but if seeing a whole log file is helpful, I can make something available. Any ideas on next steps for figuring out what’s going on? It seems like something is asking for authentication a whole lot, but it’s not clear to me what or why. We do use munge for Slurm authentication, and SSSD to work with LDAP for user authentication. -Jeremy Guillette — Jeremy Guillette Software Engineer, FAS Academic Technology | Academic Technology Harvard University Information Technology P: (617) 998-1826 | W: huit.harvard.edu (he/him/his)

4 6

Large memory jobs stuck Pending. Should use --time parameter?
by Mike 07 May '25

07 May '25

Greetings, We are new to Slurm and we are trying to better understand why we’re seeing high-mem jobs stuck in Pending state indefinitely. Smaller (mem) jobs in the queue will continue to pass by the high mem jobs even when we bump priority on a pending high-mem job way up. We have been reading over the backfill scheduling page and what we think we're seeing is that we need to require that users specify a --time parameter on their jobs so that Backfill works properly. None of our users specify a --time param because we have never required it. Is that what we need to require in order to fix this situation? From the backfill page: "Backfill scheduling is difficult without reasonable time limit estimates for jobs, but some configuration parameters that can help" and it goes on to list some config params that we have not set (DefaultTime, MaxTime, OverTimeLimit). We also see language such as, “Since the expected start time of pending jobs depends upon the expected completion time of running jobs, reasonably accurate time limits are important for backfill scheduling to work well.” So we suspect that we can achieve proper backfill scheduling by requiring that all users supply a "--time" parameter via a job submit plugin. Would that be a fair statement? Thank you in advance! -Mike Schor

4 4

2025

2024