- slurm-users - lists.schedmd.com

error building debug packages on ubuntu 24.04
by Charles Hedrick 03 Jun '25

03 Jun '25

It was unable to find lib/security/pam_slurm_adopt.so It was built as lib64/security, not lib/security. ./debian/slurm-smd-libpam-slurm-adopt.install needs to be lib64/security/pam_slurm_adopt.so More likely, it needs to be built dynamically, so it will work everywhere.

1 0

Slurm version 25.05 is now available
by Tim Wickberg 29 May '25

29 May '25

We are pleased to announce the availability of Slurm 25.05. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md The Slurm documentation has also been updated to the 25.05 release: https://slurm.schedmd.com Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

Slurm accounts managed via ansible
by Ratnasamy, Fritz 28 May '25

28 May '25

Hi, I was wondering whether there might be built in support for managing slurm accounts,users,associations in ansible. it would be nice to be able to organize accounts in a yaml style file and modify accounts settings via gitlab CI/CD. For example, in a format as such: accounts: - name: "Boss1" parent: "root" description: "Big Boos" fairshare: 20 - name: "UnderBoss1" parent: "root" description: "working for Boss1" fairshare: 1 users: -username: "user1" -username: "user2" *Fritz Ratnasamy* Data Scientist Information Technology

3 2

Account getting duplicated
by Ratnasamy, Fritz 28 May '25

28 May '25

Hi, the slurm db duplicates all our account associations. one with cluster=cluster and another where cluster=venus (which is our actual cluster). is that intended? Or should I make any changes? *Fritz Ratnasamy*Data Scientist Information Technology

2 1

Slurm Reporting Difference between sreport and sacct
by Passant Hafez 23 May '25

23 May '25

Hi all, I was wondering if someone can help explaining this discrepancy. I have different values for project gpu consumption using sreport vs sacct (+ some calculations) This is an example that shows this: sreport -t hours -T gres/gpu cluster AccountUtilizationByuser start=2025-04-01 end=2025-04-05 | grep project1234 gives 178 while sacct -n -X --allusers --accounts=project1234 --start=2025-04-01 --end=2025-04-05 -o elapsedraw,AllocTRES%80,user,partition gives 213480 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus 249507 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus 13908 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus 9552 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus 4 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu 11 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu ... I will not bore you with the full output and its calculation, but the first job alone consumed 213480 seconds/60/60 * 8 gpus that's 474.4 gpu hours which is way more than the 178 hrs reported by sreport Any clue why these are inconsistent? or how sreport calculated the 178 value? All the best, Passant

3 3

Crash in "slurmd -C" when latest NVIDIA drivers are used
by Taras Shapovalov 23 May '25

23 May '25

Hello, Does someone have idea why "slurmd -C" crashes when it unloads gpu_nrt.so with latest NVIDIA drivers (570 and 575)? We checked, there is no crash in cuda at the moment and gpu_nvml.so works fine, all nvml calls finish successfully, dlclose on gpu_nvml.so works fine. The crash does not depend whether real GPUs present or not. Steps to reproduce: 1. Install Ubuntu 24.04 2. wget https://download.schedmd.com/slurm/slurm-24.11.4.tar.bz2<https://download.schedmd.com/slurm/slurm-24.11.4.tar.bz2;> 3. tar fx ./slurm-24.11.4.tar.bz2 4. cd slurm-24.11.4 5. apt-get install cuda-12-8 hwloc libmunge-dev -y 6. ./configure 7. make && make install 8. Run "slurmd -C", or sometimes "slurmd -vvv -C" to get the crash. Stack trace: #0 0x0000155555544b2a strlen (ld-linux-x86-64.so.2 + 0x28b2a) #1 0x000015555551fc08 __GI__dl_exception_create (ld-linux-x86-64.so.2 + 0x3c08) #2 0x000015555551d298 __GI__dl_signal_error (ld-linux-x86-64.so.2 + 0x1298) #3 0x000015555551e81d _dl_close (ld-linux-x86-64.so.2 + 0x281d) #4 0x000015555551d51c __GI__dl_catch_exception (ld-linux-x86-64.so.2 + 0x151c) #5 0x000015555551d669 _dl_catch_error (ld-linux-x86-64.so.2 + 0x1669) #6 0x0000155554e97c73 _dlerror_run (libc.so.6 + 0x97c73) #7 0x0000155554e979a6 __dlclose (libc.so.6 + 0x979a6) #8 0x0000155555388a25 gpu_plugin_fini (libslurmfull.so + 0x188a25) #9 0x000015555538f2ef gres_get_autodetected_gpus (libslurmfull.so + 0x18f2ef) #10 0x0000555555564828 _print_config (slurmd + 0x10828) #11 0x0000155554e2a1ca __libc_start_call_main (libc.so.6 + 0x2a1ca) #12 0x0000155554e2a28b __libc_start_main_impl (libc.so.6 + 0x2a28b) #13 0x000055555555fc75 _start (slurmd + 0xbc75) I don't really think the problem is in gpu_nrt itself, seems the problem is in memory corruption somewhere else, but I am not sure. The issue is reproduced constantly. A Best regards, Taras

2 3

Slurm 24.11.x + pyxis + TaskProlog -> fail
by Stefan Staeglich 21 May '25

21 May '25

Hi, we are currently evaluating Slurm 24.11. We are using pyxis in our cluster and have encountered the following problem in connection with TaskProlog: https://github.com/NVIDIA/pyxis/issues/163 Do you know more about the problem and the cause? Best, Stefan

1 0

sacctmgr: error: Sending PersistInit msg: Connection refused
by Ratnasamy, Fritz 18 May '25

18 May '25

Hi, We are working on a test cluster with slurm 24.11.3 and I am getting this error message from the login or compute nodes (note that the error does not show when run from the controller node): sacctmgr list associations tree format=cluster,account,user,maxnodes sacctmgr: error: _open_persist_conn: failed to open persistent connection to host:localhost:6819: Connection refused sacctmgr: error: Sending PersistInit msg: Connection refused Does someone know how fix that issue? Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

New version of slurm
by Ratnasamy, Fritz 16 May '25

16 May '25

Hi, I am trying to install the new version of slurm. Do you know if there is a way to find out what support is compiled into the executables? For example, apache has httpd -L which shows all the loaded modules. See below result: [image: image.png] *Fritz Ratnasamy*Data Scientist Information Technology

3 2

slurmdbd - error: We can't find assoc 1 on cluster slurm
by Edmundo Carmona Antoranz 14 May '25

14 May '25

Hello! We are hitting this problem when we are making a first run of slurmdbd: [2025-05-13T13:29:08.916] debug4: accounting_storage/as_mysql: _get_parent_id: 10(as_mysql_assoc.c:699) query select id_assoc, lineage from "slurm_assoc_table" where user='' and deleted!=1 and acct='root'; [2025-05-13T13:29:08.916] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=(null)(4294967294), acct=(null), cluster=slurm, partition=(null) [2025-05-13T13:29:08.916] debug2: _find_assoc_rec_id: no associations added yet [2025-05-13T13:29:08.916] error: We can't find assoc 1 on cluster slurm I checked and up to that moment the table slurm_assoc_table has not getting a record inserted after the table has been created. [root@ea-r9-29504-pid1 ~]# grep -C1 slurm_assoc_table /var/log/slurmdbd [2025-05-13T13:29:06.666] debug2: DBD_ADD_CLUSTERS: called in CONN 10 [2025-05-13T13:29:06.719] debug4: Table "slurm_assoc_table" doesn't exist, adding [2025-05-13T13:29:06.720] debug2: query alter table "slurm_assoc_table" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default '', modify `acct` tinytext not null, modify `partition` tinytext not null default '', modify `parent_acct` tinytext not null default '', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default '', modify `max_tres_pn` text not null default '', modify `max_tres_mins_pj` text not null default '', modify `max_tres_run_mins` text not null default '', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default '', modify `grp_tres_mins` text not null default '', modify `grp_tres_run_mins` text not null default '', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default '', modify `delta_qos` blob not null default '', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42)); [2025-05-13T13:29:06.729] debug3: query insert into table_defs_table (creation_time, mod_time, table_name, definition) values (1747135746, 1747135746, '"slurm_assoc_table"', 'alter table \"slurm_assoc_table\" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default \'\', modify `acct` tinytext not null, modify `partition` tinytext not null default \'\', modify `parent_acct` tinytext not null default \'\', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default \'\', modify `max_tres_pn` text not null default \'\', modify `max_tres_mins_pj` text not null default \'\', modify `max_tres_run_mins` text not null default \'\', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default \'\', modify `grp_tres_mins` text not null default \'\', modify `grp_tres_run_mins` text not null default \'\', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default \'\', modify `delta_qos` blob not null default \'\', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42));') on duplicate key update definition='alter table \"slurm_assoc_table\" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default \'\', modify `acct` tinytext not null, modify `partition` tinytext not null default \'\', modify `parent_acct` tinytext not null default \'\', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default \'\', modify `max_tres_pn` text not null default \'\', modify `max_tres_mins_pj` text not null default \'\', modify `max_tres_run_mins` text not null default \'\', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default \'\', modify `grp_tres_mins` text not null default \'\', modify `grp_tres_run_mins` text not null default \'\', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default \'\', modify `delta_qos` blob not null default \'\', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42));', mod_time=1747135746; [2025-05-13T13:29:06.775] debug4: Table "slurm_assoc_usage_day_table" doesn't exist, adding -- [2025-05-13T13:29:08.916] debug4: accounting_storage/as_mysql: _get_parent_id: 10(as_mysql_assoc.c:699) query select id_assoc, lineage from "slurm_assoc_table" where user='' and deleted!=1 and acct='root'; [2025-05-13T13:29:08.916] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=(null)(4294967294), acct=(null), cluster=slurm, partition=(null) -- [2025-05-13T13:29:08.921] debug4: accounting_storage/as_mysql: _set_assoc_lft_rgt: 10(as_mysql_assoc.c:737) query select lft, rgt from "slurm_assoc_table" where id_assoc=2; [2025-05-13T13:29:08.922] debug4: accounting_storage/as_mysql: _make_sure_user_has_default_internal: 10(as_mysql_assoc.c:401) query select distinct is_def, acct from "slurm_assoc_table" where user='root' and deleted!=1 ORDER BY is_def desc, creation_time desc LIMIT 1; [2025-05-13T13:29:08.924] debug4: accounting_storage/as_mysql: as_mysql_add_assocs: 10(as_mysql_assoc.c:3999) query Even though the message shows up in the logs as an _error_, the service is able to come up fine, by the way, but is there an easy way to avoid the error message in the first place? BR!

1 0

2025

2024