- slurm-users - lists.schedmd.com

New version of slurm
by Ratnasamy, Fritz 16 May '25

16 May '25

Hi, I am trying to install the new version of slurm. Do you know if there is a way to find out what support is compiled into the executables? For example, apache has httpd -L which shows all the loaded modules. See below result: [image: image.png] *Fritz Ratnasamy*Data Scientist Information Technology

3 2

slurmdbd - error: We can't find assoc 1 on cluster slurm
by Edmundo Carmona Antoranz 14 May '25

14 May '25

Hello! We are hitting this problem when we are making a first run of slurmdbd: [2025-05-13T13:29:08.916] debug4: accounting_storage/as_mysql: _get_parent_id: 10(as_mysql_assoc.c:699) query select id_assoc, lineage from "slurm_assoc_table" where user='' and deleted!=1 and acct='root'; [2025-05-13T13:29:08.916] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=(null)(4294967294), acct=(null), cluster=slurm, partition=(null) [2025-05-13T13:29:08.916] debug2: _find_assoc_rec_id: no … [View More]associations added yet [2025-05-13T13:29:08.916] error: We can't find assoc 1 on cluster slurm I checked and up to that moment the table slurm_assoc_table has not getting a record inserted after the table has been created. [root@ea-r9-29504-pid1 ~]# grep -C1 slurm_assoc_table /var/log/slurmdbd [2025-05-13T13:29:06.666] debug2: DBD_ADD_CLUSTERS: called in CONN 10 [2025-05-13T13:29:06.719] debug4: Table "slurm_assoc_table" doesn't exist, adding [2025-05-13T13:29:06.720] debug2: query alter table "slurm_assoc_table" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default '', modify `acct` tinytext not null, modify `partition` tinytext not null default '', modify `parent_acct` tinytext not null default '', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default '', modify `max_tres_pn` text not null default '', modify `max_tres_mins_pj` text not null default '', modify `max_tres_run_mins` text not null default '', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default '', modify `grp_tres_mins` text not null default '', modify `grp_tres_run_mins` text not null default '', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default '', modify `delta_qos` blob not null default '', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42)); [2025-05-13T13:29:06.729] debug3: query insert into table_defs_table (creation_time, mod_time, table_name, definition) values (1747135746, 1747135746, '"slurm_assoc_table"', 'alter table \"slurm_assoc_table\" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default \'\', modify `acct` tinytext not null, modify `partition` tinytext not null default \'\', modify `parent_acct` tinytext not null default \'\', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default \'\', modify `max_tres_pn` text not null default \'\', modify `max_tres_mins_pj` text not null default \'\', modify `max_tres_run_mins` text not null default \'\', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default \'\', modify `grp_tres_mins` text not null default \'\', modify `grp_tres_run_mins` text not null default \'\', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default \'\', modify `delta_qos` blob not null default \'\', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42));') on duplicate key update definition='alter table \"slurm_assoc_table\" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default \'\', modify `acct` tinytext not null, modify `partition` tinytext not null default \'\', modify `parent_acct` tinytext not null default \'\', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default \'\', modify `max_tres_pn` text not null default \'\', modify `max_tres_mins_pj` text not null default \'\', modify `max_tres_run_mins` text not null default \'\', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default \'\', modify `grp_tres_mins` text not null default \'\', modify `grp_tres_run_mins` text not null default \'\', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default \'\', modify `delta_qos` blob not null default \'\', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42));', mod_time=1747135746; [2025-05-13T13:29:06.775] debug4: Table "slurm_assoc_usage_day_table" doesn't exist, adding -- [2025-05-13T13:29:08.916] debug4: accounting_storage/as_mysql: _get_parent_id: 10(as_mysql_assoc.c:699) query select id_assoc, lineage from "slurm_assoc_table" where user='' and deleted!=1 and acct='root'; [2025-05-13T13:29:08.916] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=(null)(4294967294), acct=(null), cluster=slurm, partition=(null) -- [2025-05-13T13:29:08.921] debug4: accounting_storage/as_mysql: _set_assoc_lft_rgt: 10(as_mysql_assoc.c:737) query select lft, rgt from "slurm_assoc_table" where id_assoc=2; [2025-05-13T13:29:08.922] debug4: accounting_storage/as_mysql: _make_sure_user_has_default_internal: 10(as_mysql_assoc.c:401) query select distinct is_def, acct from "slurm_assoc_table" where user='root' and deleted!=1 ORDER BY is_def desc, creation_time desc LIMIT 1; [2025-05-13T13:29:08.924] debug4: accounting_storage/as_mysql: as_mysql_add_assocs: 10(as_mysql_assoc.c:3999) query Even though the message shows up in the logs as an _error_, the service is able to come up fine, by the way, but is there an easy way to avoid the error message in the first place? BR! [View Less]

1 0

Slurm release candidate version 25.05.0rc1 is available for testing
by Marshall Garey 13 May '25

13 May '25

We are pleased to announce the availability of Slurm release candidate 25.05.0rc1. To highlight some new features coming in 25.05: - Support for defining multiple topology configurations, and varying them by partition. - Support for tracking and allocating hierarchical resources. - Dynamic nodes can be dynamically added to the topology. - topology/block - Allow for gaps in the block layout. - Support for encrypting all network communication with TLS. - jobcomp/kafka - Optionally send job … [View More]

1 0

Performance Issues after Update to 24.11.5
by Tilman Hoffbauer 13 May '25

13 May '25

Hello, we are running a SLURM-managed cluster with one control node (g-vm03) and 26 worker nodes (ouga[03-28]) on Rocky 8. We recently updated from 20.11.9 through 23.02.8 to 24.11.0 and then 24.11.5. Since then, we are experiencing performance issues - squeue and scontrol ping are slow to react and sometimes deliver "timeout on send/recv" messages, even with only very few parallel requests. We did not experience these issues with SLURM 20.11.9 before, we did not check the intermediate … [View More]

2 4

Do I have to hold back RAM for worker nodes?
by Xaver Stiensmeier 13 May '25

13 May '25

Dear Slurm-User List, currently, in our slurm.conf, we are setting: SelectType: select/cons_tres SelectTypeParameters: CR_Core and in our node configuration /RealMemory /was basically reduced by an amount to make sure the node always had enough RAM to run the OS. However, this is apparently now how it is supposed to be done: Lowering RealMemory with the goal of setting aside some amount for the OS and not available for job allocations will not work as intended if … [View More]

6 8

slurm failing to start as it doesnt know its node name??????
by Steven Jones 12 May '25

12 May '25

Hi, I am not sure what i have missed but I am getting this error on a compute node. ====== root@vm01no16 ~]# systemctl status slurmd.service × slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Sun 2025-05-11 23:01:29 UTC; 15min ago Process: 1178 ExecStart=/usr/sbin/slurmd --systemd $SLURMD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 1178 (code=exited, status=1/… [View More]

2 1

Slurm versions 24.11.5, 24.05.8, and 23.11.11 are now available (CVE-2025-43904)
by Tim Wickberg 07 May '25

07 May '25

Slurm versions 24.11.5, 24.05.8, and 23.11.11 are now available and include a fix for a recently discovered security issue. SchedMD customers were informed on April 23rd and provided a patch on request; this process is documented in our security policy. [1] A mistake with permission handling for Coordinators within Slurm's accounting system can allow a Coordinator to promote a user to Administrator. (CVE-2025-43904) Thank you to Sekou Diakite (HPE) for reporting this. Downloads are … [View More]available at https://www.schedmd.com/downloads.php . Release notes follow below. - Tim [1] https://www.schedmd.com/security-policy/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support > * Changes in Slurm 24.11.5 > ========================== > -- Return error to scontrol reboot on bad nodelists. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.40 > endpoints. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.41 > endpoints. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.42 > endpoints. > -- data_parser/v0.0.42 - Added +inline_enums flag which modifies the > output when generating OpenAPI specification. It causes enum arrays to not > be defined in their own schema with references ($ref) to them. Instead they > will be dumped inline. > -- Fix binding error with tres-bind map/mask on partial node allocations. > -- Fix stepmgr enabled steps being able to request features. > -- Reject step creation if requested feature is not available in job. > -- slurmd - Restrict listening for new incoming RPC requests further into > startup. > -- slurmd - Avoid auth/slurm related hangs of CLI commands during startup > and shutdown. > -- slurmctld - Restrict processing new incoming RPC requests further into > startup. Stop processing requests sooner during shutdown. > -- slurmcltd - Avoid auth/slurm related hangs of CLI commands during > startup and shutdown. > -- slurmctld: Avoid race condition during shutdown or reconfigure that > could result in a crash due delayed processing of a connection while > plugins are unloaded. > -- Fix small memleak when getting the job list from the database. > -- Fix incorrect printing of % escape characters when printing stdio > fields for jobs. > -- Fix padding parsing when printing stdio fields for jobs. > -- Fix printing %A array job id when expanding patterns. > -- Fix reservations causing jobs to be held for Bad Constraints > -- switch/hpe_slingshot - Prevent potential segfault on failed curl > request to the fabric manager. > -- Fix printing incorrect array job id when expanding stdio file names. > The %A will now be substituted by the correct value. > -- Fix printing incorrect array job id when expanding stdio file names. > The %A will now be substituted by the correct value. > -- switch/hpe_slingshot - Fix vni range not updating on slurmctld restart > or reconfigre. > -- Fix steps not being created when using certain combinations of -c and > -n inferior to the jobs requested resources, when using stepmgr and nodes > are configured with CPUs == Sockets*CoresPerSocket. > -- Permit configuring the number of retry attempts to destroy CXI service > via the new destroy_retries SwitchParameter. > -- Do not reset memory.high and memory.swap.max in slurmd startup or > reconfigure as we are never really touching this in slurmd. > -- Fix reconfigure failure of slurmd when it has been started manually and > the CoreSpecLimits have been removed from slurm.conf. > -- Set or reset CoreSpec limits when slurmd is reconfigured and it was > started with systemd. > -- switch/hpe-slingshot - Make sure the slurmctld can free step VNIs after > the controller restarts or reconfigures while the job is running. > -- Fix backup slurmctld failure on 2nd takeover. > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904. > * Changes in Slurm 24.05.8 > ========================== > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904. > * Changes in Slurm 23.11.11 > =========================== > -- Fixed a job requeuing issue that merged job entries into the same SLUID > when all nodes in a job failed simultaneously. > -- Add ABORT_ON_FATAL environment variable to capture a backtrace from any > fatal() message. > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904. [View Less]

1 0

Lots of RPC calls and REQUEST_GETPW calls
by Guillette, Jeremy 07 May '25

07 May '25

Hello, I’m trying to figure out why we’ve been seeing an increase in network traffic in our AWS-based cluster, which uses Amazon’s parallel cluster tool. After an incident a couple weeks ago, I turned on debug2 logging on the slurmd processes, and I’m seeing huge numbers of `REQUEST_GETPW` and `REQUEST_GETGR` requests going to the slurmd processes. I briefly turned on debug2 logging for `slurmctld` as well, and I’m seeing lots of RPC calls, but not as many as the `REQUEST_GETPW` requests that I’… [View More]ve seen on compute node slurmd processes. Here’s a sample from the slurmctld log: ``` [2025-04-28T15:11:05.436] debug2: _slurm_rpc_dump_partitions, size=1253 usec=20 [2025-04-28T15:11:05.450] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2971 [2025-04-28T15:11:05.451] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2971 [2025-04-28T15:11:05.451] debug2: _slurm_rpc_dump_partitions, size=1253 usec=16 [2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2788 [2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2788 [2025-04-28T15:11:05.461] debug2: _slurm_rpc_dump_partitions, size=1253 usec=12 [2025-04-28T15:11:05.517] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2916 [2025-04-28T15:11:05.518] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2916 [2025-04-28T15:11:05.518] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.628] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3405 [2025-04-28T15:11:05.629] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3405 [2025-04-28T15:11:05.629] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2189 [2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2189 [2025-04-28T15:11:05.740] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:05.845] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2209 [2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2209 [2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=4106 [2025-04-28T15:11:05.846] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.847] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=4106 [2025-04-28T15:11:05.847] debug2: _slurm_rpc_dump_partitions, size=1253 usec=11 [2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3400 [2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3400 [2025-04-28T15:11:05.938] debug2: _slurm_rpc_dump_partitions, size=1253 usec=18 [2025-04-28T15:11:06.903] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3449 [2025-04-28T15:11:06.904] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3449 [2025-04-28T15:11:06.904] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.175] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3722 [2025-04-28T15:11:07.176] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3722 [2025-04-28T15:11:07.177] debug2: _slurm_rpc_dump_partitions, size=1253 usec=254 [2025-04-28T15:11:07.205] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=4040 [2025-04-28T15:11:07.206] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=4040 [2025-04-28T15:11:07.206] debug2: _slurm_rpc_dump_partitions, size=1253 usec=17 [2025-04-28T15:11:07.237] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2990 [2025-04-28T15:11:07.238] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2990 [2025-04-28T15:11:07.239] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.284] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2920 [2025-04-28T15:11:07.285] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2920 [2025-04-28T15:11:07.285] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.370] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3236 [2025-04-28T15:11:07.371] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3236 [2025-04-28T15:11:07.371] debug2: _slurm_rpc_dump_partitions, size=1253 usec=17 [2025-04-28T15:11:08.463] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2848 [2025-04-28T15:11:08.464] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2848 [2025-04-28T15:11:08.464] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:08.691] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2627 [2025-04-28T15:11:08.692] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2627 [2025-04-28T15:11:08.692] debug2: _slurm_rpc_dump_partitions, size=1253 usec=18 [2025-04-28T15:11:08.873] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3729 [2025-04-28T15:11:08.874] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3729 [2025-04-28T15:11:08.875] debug2: _slurm_rpc_dump_partitions, size=1253 usec=196 [2025-04-28T15:11:08.881] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3461 [2025-04-28T15:11:08.882] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3461 [2025-04-28T15:11:08.882] debug2: _slurm_rpc_dump_partitions, size=1253 usec=10 ``` And from slurmd: ``` [2025-04-27T19:45:01.353] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.475] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.491] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.501] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.504] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.507] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.513] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.606] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.607] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:04.988] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.992] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.995] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.999] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.011] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.016] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.033] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.045] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.048] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.057] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.073] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.077] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.110] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.144] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.170] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.172] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.174] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.203] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.204] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.207] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.316] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.318] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.321] [59253.batch] debug: Handling REQUEST_GETPW ``` This level of debugging makes the logs pretty huge, but if seeing a whole log file is helpful, I can make something available. Any ideas on next steps for figuring out what’s going on? It seems like something is asking for authentication a whole lot, but it’s not clear to me what or why. We do use munge for Slurm authentication, and SSSD to work with LDAP for user authentication. -Jeremy Guillette — Jeremy Guillette Software Engineer, FAS Academic Technology | Academic Technology Harvard University Information Technology P: (617) 998-1826 | W: huit.harvard.edu (he/him/his) [View Less]

4 6

Large memory jobs stuck Pending. Should use --time parameter?
by Mike 07 May '25

07 May '25

Greetings, We are new to Slurm and we are trying to better understand why we’re seeing high-mem jobs stuck in Pending state indefinitely. Smaller (mem) jobs in the queue will continue to pass by the high mem jobs even when we bump priority on a pending high-mem job way up. We have been reading over the backfill scheduling page and what we think we're seeing is that we need to require that users specify a --time parameter on their jobs so that Backfill works properly. None of our users specify … [View More]

4 4

Trying to using PMIx via srun for openmpi + pmix 5.0.7, PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 (but works with --mpi=pmix_v3?!)
by Dr. Thomas Orgis 06 May '25

06 May '25

Dear Slurm community, I am confused by the behaviour of a freshly built openmpi-5.0.7 with slurm-24.11.4. I can run a simple hello-world program via mpirun, but with really slow startup (a single process needing 1.6 s, 384 processes on two 192-core nodes need around half a minute). I guess there is a deeper issue here to work out with openmpi itself and why it needs 1.3 seconds to start a single process, even, outside a Slurm environment. So I investigate if running via srun changes things. … [View More]Open-MPI docs recommend using mpirun and that has been traditionally our safe bet. But srun direct launch is supposed to work, too. I did duly build slurm-24.11.4 with ./configure --prefix=/syssw/slurm/24.11.4 \ --sysconfdir=/syssw/etc/slurm \ --with-munge=/syssw/munge/0.5.16 \ --with-hwloc=/syssw/hwloc/2.11.2 \ --disable-static --with-json \ --with-pmix=/syssw/pmix/3.2.5:/syssw/pmix/5.0.7 \ LDFLAGS=-Wl,--disable-new-dtags providing the two versions of pmix on our system currently. Now I am perplexed to observe that $ srun -vv --mpi=pmix_v5 -N 1 -n 1 mpihello does _not_ work, but produces c$ srun -vv -n 1 -N 1 --mpi=pmix_v5 mpihello srun: defined options srun: -------------------- -------------------- srun: (null) : n[164-165] srun: jobid : 671133 srun: job-name : interactive srun: mpi : pmix_v5 srun: nodes : 1 srun: ntasks : 1 srun: verbose : 2 srun: -------------------- -------------------- srun: end of defined options srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0022 srun: jobid 671133: nodes(2):`n[164-165]', cpu counts: 192(x2) srun: debug: requesting job 671133, user 99, nodes 1 including ((null)) srun: debug: cpus 1, tasks 1, name mpihello, relative 65534 srun: CpuBindType=none srun: debug: Entering slurm_step_launch srun: debug: mpi/pmix_v5: pmixp_abort_agent_start: (null) [0]: pmixp_agent.c:417: Abort agent port: 36505 srun: debug: mpi/pmix_v5: _pmix_abort_thread: (null) [0]: pmixp_agent.c:356: Start abort thread srun: debug: mpi/pmix_v5: mpi_p_client_prelaunch: (null) [0]: mpi_pmix.c:282: setup process mapping in srun srun: debug: Entering _msg_thr_create() srun: debug: initialized stdio listening socket, port 38681 srun: debug: Started IO server thread srun: debug: Entering _launch_tasks srun: launching StepId=671133.3 on host n165, 1 tasks: 0 srun: topology/tree: init: topology tree plugin loaded srun: debug: launch returned msg_rc=0 err=0 type=8001 srun: Node n165, 1 tasks started [n165:2287661] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and MPI will try to terminate your MPI job as well) [n165:2287661] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! srun: Received task exit notification for 1 task of StepId=671133.3 (status=0x0e00). srun: error: n165: task 0: Exited with exit code 14 srun: debug: task 0 done srun: debug: IO thread exiting srun: debug: mpi/pmix_v5: _conn_readable: (null) [0]: pmixp_agent.c:109: false, shutdown srun: debug: mpi/pmix_v5: _pmix_abort_thread: (null) [0]: pmixp_agent.c:363: Abort thread exit Observe the line [n165:2287661] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 anyone got an idea what that means, by what it's caused? And, what really confuses me, is that the test program _does_ work if I switch from pmix_v5 to pmix_v3: $ srun -vv -n 1 -N 1 --mpi=pmix_v3 mpihello srun: defined options srun: -------------------- -------------------- srun: (null) : n[164-165] srun: jobid : 671133 srun: job-name : interactive srun: mpi : pmix_v3 srun: nodes : 1 srun: ntasks : 1 srun: verbose : 2 srun: -------------------- -------------------- srun: end of defined options srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0022 srun: jobid 671133: nodes(2):`n[164-165]', cpu counts: 192(x2) srun: debug: requesting job 671133, user 99, nodes 1 including ((null)) srun: debug: cpus 1, tasks 1, name mpihello, relative 65534 srun: CpuBindType=none srun: debug: Entering slurm_step_launch srun: debug: mpi/pmix_v3: pmixp_abort_agent_start: (null) [0]: pmixp_agent.c:417: Abort agent port: 43737 srun: debug: mpi/pmix_v3: _pmix_abort_thread: (null) [0]: pmixp_agent.c:356: Start abort thread srun: debug: mpi/pmix_v3: mpi_p_client_prelaunch: (null) [0]: mpi_pmix.c:282: setup process mapping in srun srun: debug: Entering _msg_thr_create() srun: debug: initialized stdio listening socket, port 43805 srun: debug: Started IO server thread srun: debug: Entering _launch_tasks srun: launching StepId=671133.4 on host n164, 1 tasks: 0 srun: topology/tree: init: topology tree plugin loaded srun: debug: launch returned msg_rc=0 err=0 type=8001 srun: Node n164, 1 tasks started hello world from processor n164, rank 0 out of 1 srun: Received task exit notification for 1 task of StepId=671133.4 (status=0x0000). srun: n164: task 0: Completed srun: debug: task 0 done srun: debug: IO thread exiting srun: debug: mpi/pmix_v3: _conn_readable: (null) [0]: pmixp_agent.c:109: false, shutdown srun: debug: mpi/pmix_v3: _pmix_abort_thread: (null) [0]: pmixp_agent.c:363: Abort thread exit How can a PMIx 5 MPI even work with the pmix_v3 plugin? Why does it _not_ work with the pmix_v5 plugin? I am also curious why the plugins don't link to the respective libpmix (are they using dlopen for their dependencies? Why?). $ ldd /syssw/slurm/24.11.4/lib/slurm/mpi_pmix*.so /syssw/slurm/24.11.4/lib/slurm/mpi_pmix.so: linux-vdso.so.1 (0x00007ffd19ffb000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x000014d199e56000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014d199c70000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014d199b90000) /lib64/ld-linux-x86-64.so.2 (0x000014d199edd000) /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v3.so: linux-vdso.so.1 (0x00007ffd265f2000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x00001553902c8000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00001553900e2000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000155390002000) /lib64/ld-linux-x86-64.so.2 (0x0000155390350000) /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v5.so: linux-vdso.so.1 (0x00007ffd862b7000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x0000145adc36d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000145adc187000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000145adc0a7000) /lib64/ld-linux-x86-64.so.2 (0x0000145adc3f4000) But they do have the proper RPATH set up: $ readelf -d /syssw/slurm/24.11.4/lib/slurm/mpi_pmix*.so | grep -e ^File -e PATH File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/5.0.7/lib] File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v3.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/3.2.5/lib] File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v5.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/5.0.7/lib] Which is important, since libpmix doesn't get sensible SONAME versioning (supposing they are supposed to be separate ABIs): $ find /syssw/pmix/* -name 'libpmix.so*' /syssw/pmix/3.2.5/lib/libpmix.so /syssw/pmix/3.2.5/lib/libpmix.so.2.2.35 /syssw/pmix/3.2.5/lib/libpmix.so.2 /syssw/pmix/5.0.7/lib/libpmix.so /syssw/pmix/5.0.7/lib/libpmix.so.2.13.7 /syssw/pmix/5.0.7/lib/libpmix.so.2 It's all libpmix.so.2. My mpihello program uses the 5.0.7 one, at least: $ ldd mpihello linux-vdso.so.1 (0x00007fff1f13f000) libmpi.so.40 => /sw/env/gcc-13.3.0/openmpi/5.0.7/lib/libmpi.so.40 (0x000014ed89d95000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014ed89baf000) libopen-pal.so.80 => /sw/env/gcc-13.3.0/openmpi/5.0.7/lib/libopen-pal.so.80 (0x000014ed89a25000) libfabric.so.1 => /syssw/fabric/1.21.0/lib/libfabric.so.1 (0x000014ed8989b000) libefa.so.1 => /lib/x86_64-linux-gnu/libefa.so.1 (0x000014ed8988d000) libibverbs.so.1 => /lib/x86_64-linux-gnu/libibverbs.so.1 (0x000014ed8986c000) libpsm2.so.2 => /syssw/psm2/12.0.1/lib/libpsm2.so.2 (0x000014ed89804000) libatomic.so.1 => /sw/compiler/gcc-13.3.0/lib64/libatomic.so.1 (0x000014ed897fb000) libnl-route-3.so.200 => /lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x000014ed8976a000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x000014ed89745000) libpmix.so.2 => /syssw/pmix/5.0.7/lib/libpmix.so.2 (0x000014ed8951e000) libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x000014ed894e8000) libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x000014ed894e3000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x000014ed89486000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014ed893a6000) /lib64/ld-linux-x86-64.so.2 (0x000014ed8a0d8000) libnuma.so.1 => /lib/x86_64-linux-gnu/libnuma.so.1 (0x000014ed89397000) Can someone shed light on how the differing PMIx plugins are supposed to work? Can someone share a setup where pmix_v5 does work with openmpi 5.x? Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg [View Less]

1 0

knl_generic.conf seems to be ignored and Slurm adopts default settings which specify Intel motherboard and syscfg - need help to figure out why Slurm is not reading my Dell knl_generic.conf parameters
by Bryan Johnston 01 May '25

01 May '25

Hi folks I'm in the process of standing up some Stampede2 Dell C3620p KNL nodes and I seem to be hitting a blind spot. I previously "successfully" configured KNL's on an Intel board (S7200AP), with OpenHPC and Rocky8. I say "successfully" because it works but evidently my latest troubleshooting has revealed that I may have been lucky rather than an expert KNL integrator :-) I thought I knew what I was doing, but after repeating my Intel KNL recipe with the Dell system, I have unearthed my … [View More]ignorance with this wonderful (but deprecated) technology (anecdotally, the KNLs offer excellent performance and power efficiency for our workloads, particularly when contrasted with our alternative available hardware). The first discovery was the "syscfg" for Intel boards is not the same as the "syscfg" for Dell boards. I've since sorted this out. The second discovery was made while troubleshooting an issue that I'm hitting. After realising that the slurmd client nodes don't seem to be reading the "knl_generic.conf" parameters that are specified in /etc/slurm on the smshost (OpenHPC parlance for head node ... And it's a Slurm config less set up), I think my original Intel solution was working out of luck more than ingenuity. For reference , the Slurm configuration for KNL now includes: ``` NodeFeaturesPlugins=knl_generic DebugFlags=NodeFeatures GresTypes=hbm ``` And I've created a separate "knl_generic.conf" that points to the Dell specific tools and features. For the Dell system, slurmd seems to ignore my knl_generic.conf file and is drawing defaults from somewhere else. Slurm still considers SystemType to be Intel, SyscfgPath to be the default location, and SyscfgTimeout to be 1000. For Dell systems, Slurm needs to have SystemType=Dell and Timeout to be 1000. I don't understand why the nodes are not reading the knl_generic file - any help or clues would be appreciated. Here's my theory on what is happening: The Intel KNL system was successful by luck ... It probably exhibited the same ignore-the-config-file but ran default NodeFeatures for some generic knl_generic settings which are stored somewhere as default parameters. I must have just lucked out when I was using my Intel KNL system because it was using the defaults (that are compatible with Intel). If this assumption is correct, the Dell system is not working because it isn't compatible with the Intel defaults. Any clues on how to successfully invoke the config file (or better debuggingtechniwues to figure out why it isn't) would be appreciated. I can share journalctl feedback if necessary. For now, I've tried changing ownership of the config files to root:slurm, copied knl_generic.conf to the compute nodes' /etc/slurm/ and also tried to specify the config file by running (on the compute nodes) "slurmd" with "-f" ... No joy; if slurmd runs successfully (when I don't screw up some random experimental settings) then it always seems to ignore knl_generic.conf and loads some default settings from somewhere. A few questions: 1. Are there default settings stored somewhere? I might be barking up the wrong tree, although I've looked for files that may clash with the config file I've created but can't find any. 2. Is there a better way to force the knl_generic file to be loaded? 3. Is the configless Slurm somehow not reading the knl_generic file to the clients? I understand that all configuration files are read from the host server. Many thanks for any help! Regards / Groete / Sala(ni) Kahle - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Bryan Johnston Senior HPC Technologist II Lead: HPC Ecosystems Project HPCwire 2024's Outstanding Leader in HPC CHPC | www.chpc.ac.za | NICIS | nicis.ac.za Centre for High Performance Computing If you receive an email from me out of office hours for you, please do not feel obliged to respond during off-hours! Book time to meet with me<https://outlook.office.com/bookwithme/user/87af4906a703488386578f34e4473c74…> [View Less]

1 1

Slurm and Kerberos/GSSAPI
by Burian, John 01 May '25

01 May '25

Does anyone have any experience with using Kerberos/GSSAPI and Slurm? I’m specifically wondering if there is a known mechanism for providing proper Kerberos credentials to Slurm batch jobs, such that those processes would be able to access a filesystem that requires Kerberos credentials. Some quick searching returned nothing useful. Interactive jobs have a similar problem, but I’m hoping that SSH credential forwarding can be leveraged there. I’m nothing like an expert in Kerberos, so forgive … [View More]

2 3

Can't specify multiple partitions when submitting GPU jobs
by milad＠earthspecies.org 29 Apr '25

29 Apr '25

Hi, I have defined a partition for each GPU type we have in the cluster. This was mainly because I have different Node types for each GPU type and I want to set `DefCpuPerGPU` `DefMemPerGPU` for each of them. Unfortunately one can't set them per node but can do that per partition. Now sometimes people don't care about the GPU type and would like any of the partitions to pick up the job. The `--partition` in `sbatch` does allow specifying multiple partitions and this works fine when I'm not … [View More]

3 4

Re: Error "_check_core_range_matches_sock" when
by Gestió Servidors 29 Apr '25

29 Apr '25

Hello, Yes, I know system is single-socket, but in a similar computer, this configuration works fine... After some tests, I have realized that with SLURM 23.11.0 I can assign CPUs in "gres.conf" (and, then, use half CPU for one GPU and other half for other GPU), but with SLURM 24.11.4 I receive " _check_core_range_matches_sock". I have read "NEWS" file inside sources files in 24.11.4 tgz and it says: […] * Changes in Slurm 23.11.7 ========================== […] -- Fix issue that only … [View More]

1 0

Error "_check_core_range_matches_sock" when starting "slurmctld"
by Gestió Servidors 28 Apr '25

28 Apr '25

Hello, I have compiled SLURM-24.11.3 and I have configured two GPUs in my system (slurmctld and slurmd are running in the same computer). Computes has a old processor Intel i7 with 4 cores and 4 hyperthreading. Node is configured with "NodeName=mysystem CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7940 Gres=gpu:geforce_gtx_titan_x:1,gpu:geforce_gtx_titan_black:1". "lscpu" command returns: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte … [View More]

2 1

How can we put limits on interactive jobs?
by Ole Holm Nielsen 25 Apr '25

25 Apr '25

We would like to put limits on interactive jobs (started by salloc) so that users don't leave unused interactive jobs behind on the cluster by mistake. I can't offhand find any configurations that limit interactive jobs, such as enforcing a timelimit. Perhaps this could be done in job_submit.lua, but I couldn't find any job_desc parameters in the source code which would indicate if a job is interactive or not. Question: How do people limit interactive jobs, or identify orphaned jobs … [View More]

4 4

Setting QoS with slurm 24.05.7
by Patrick Begou 25 Apr '25

25 Apr '25

Hi all, I'm trying to setup a QoS on a small 5 nodes cluster running slurm 24.05.7. My goal is to limit the resources on a (time x number of cores) strategy to avoid one large job requesting all the resources for too long time. I've read from https://slurm.schedmd.com/qos.html and some discussion but my setup is still not working. I think I need to set these informations: MaxCPUsPerJob=172800 MaxWallDurationPerJob=48:00:00 Flags=DenyOnLimit,OverPartQOS for: 12h max for 240 cores => (… [View More]12*240*60=172800mn) no job can exceed 2 days do not accept jobs out of these limits. What I've done: 1) create the QoS: sudo sacctmgr add qos workflowlimit \ MaxWallDurationPerJob=48:00:00 \ MaxCPUsPerJob=172800 \ Flags=DenyOnLimit,OverPartQOS 2) Check sacctmgr show qos Name=workflowlimit format=Name%16,MaxTRES,MaxWall Name MaxTRES MaxWall ---------------- ------------- ----------- workflowlimit cpu=172800 2-00:00:00 3) Set the QoS for the account "most" which is the default account for the users: sudo sacctmgr modify account name=most set qos=workflowlimit 4) Check $ sacctmgr show assoc format=account,cluster,user,qos Account Cluster User QOS ---------- ---------- ---------- -------------------- root osorno normal root osorno root normal legi osorno normal most osorno workflowlimit most osorno begou workflowlimit 5) Modifiy slurm.conf with: AccountingStorageEnforce=limits,qos and propagate on the 5 nodes and the front end (done via Ansible) 6) Check clush -b -w osorno-fe,osorno,osorno-0-[0-4] 'grep AccountingStorageEnforce /etc/slurm/slurm.conf' --------------- osorno,osorno-0-[0-4],osorno-fe (7) --------------- AccountingStorageEnforce=limits,qos 7) restart slurmd on all the compute nodes and slurmctld + slurmdbd on the management node. But I can still request 400 cores for 24 hours: [begou@osorno ~]$ srun -n 400 -t 24:0:0 --pty bash bash-5.1$ squeue JOBID PARTITION NAME USER ST TIME START_TIME TIME_LIMIT CPUS NODELIST(REASON) 147 genoa bash begou R 0:03 2025-04-18T16:52:11 1-00:00:00 400 osorno-0-[0-4] So I must have missed something ? My partition (I've only one) in slurm.conf is: PartitionName=genoa State=UP Default=YES MaxTime=48:00:00 DefaultTime=24:00:00 Shared=YES OverSubscribe=NO Nodes=osorno-0-[0-4] Thanks Patrick [View Less]

2 4

Job running slower when using Slurm
by Jeffrey Layton 24 Apr '25

24 Apr '25

Good morning, I'm running an NPB test, bt.C that is OpenMP and built using NV HPC SDK (version 25.1). I run it on a compute node by ssh-ing to the node. It runs in about 19.6 seconds. Then I run the code using a simple job: Command to submit job: sbatch --nodes=1 run-npb-omp The script run-npb-omp is the following: #!/bin/bash cd /home/.../NPB3.4-OMP/bin ./bt.C.x When I use Slurm, the job takes 482 seconds. Nothing really appears in the logs. It doesn't do any IO. No data is copied … [View More]

4 9

Strange output of sshare
by frsc＠adm.ku.dk 23 Apr '25

23 Apr '25

Hi all, I'm trying to clean up and reconfigure fair share on a slurm 20.11.9 production cluster after some trial and error before I started looking into it. I don't know the full story and need to pick up here. Fair share is enabled with default settings and not customised. It looks a bit like it was enabled by accident and relevant options are undefined/at default. As a first step I would like to understand the current state. Executing sshare includes output like this (output shortened but … [View More]complete for the two relevant accounts): Account User Partition RawShares NormShares RawUsage EffectvUsage FairShare -------- ---------- ------------ ---------- ----------- ----------- ------------- ---------- root 0.000000 1002501 1.000000 .root root 1 0.083333 0 0.000000 1.000000 .A1 1 0.083333 0 0.000000 ..A1 U1 P1 1 0.166667 0 0.000000 0.873684 ..A1 U2 P1 1 0.166667 0 0.000000 0.873684 ..A1 U3 P1 1 0.166667 0 0.000000 0.873684 ..A1 U3 P2 1 0.166667 0 0.000000 0.873684 ..A1 U4 P3 1 0.166667 0 0.000000 0.873684 ..A1 U4 P1 1 0.166667 0 1.000000 0.821053 <== What is going on here?? .A2 1 0.083333 0 0.000000 ..A2 U5 P2 1 0.142857 0 0.000000 1.000000 ..A2 U5 P1 1 0.142857 0 0.000000 1.000000 ..A2 U6 P4 1 0.142857 0 0.000000 1.000000 ..A2 U6 P1 1 0.142857 0 0.000000 1.000000 ..A2 U6 P2 1 0.142857 0 0.000000 1.000000 ..A2 U7 P1 1 0.142857 0 0.000000 1.000000 ..A2 U7 P2 1 0.142857 0 0.000000 1.000000 User U4 is not a member of any other account. I understand everything about this output except the line I marked. Both accounts A1 and A2 have zero usage, yet, for user U4 on partition P1 we have effective usage 1.0, screwing the fair share factors up for everyone in this account. As far as I can tell, both accounts should look identical with a fair share factor of 1 for every user. I'm grateful for any pointer for what to look for. Best regards, Frank [View Less]

2 1

Slurm webhooks
by Davide DelVento 22 Apr '25

22 Apr '25

Happy Monday everybody, I've gotten a request to have Slurm notify users for the typical email things (job started, completed, failed, etc) with a REST API instead of email. This would allow notifications in MS Teams, Slack, or log stuff in some internal websites and things like that. As far as I can tell, Slurm does not support that, for example there was somebody who was looking for that on Galaxy and did not find a solution: https://help.galaxyproject.org/t/web-hook-post-to-external-url-… [View More]

3 3

problem with slurmd dynamic mode with slurmctld in docker
by stef53864 22 Apr '25

22 Apr '25

hi everybody, i try to user dynamic mode with configless mode with slurm 24.11.3 and upgrade to 24.11.4 with slurmd and i found a problem. slurmctld is a container with docker, and my node is outside the container network . slurmctld register my ip with a function getpeeraddr on the slurmctld socket. but my ip connected to the socket come from the docker nat/bridge so slurmctld register my ip bridged ( not my real ip ) that is to say the docker gateway (172.20.0.1) *scontrol show node* *… [View More]

1 0

pam_slurm_adopt ambiguity in instructions and with ssd
by Robert Kudyba 21 Apr '25

21 Apr '25

In the instructions for pam_slurm_adopt <https://slurm.schedmd.com/pam_slurm_adopt.html#ssh_config>, there are instructions such as: > > Add the following line to the appropriate file in /etc/pam.d, such as > system-auth or sshd (you may use either the "required" or "sufficient" > PAM control flag): This module is configurable. Add these options to the end of the > pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd > or system-auth): Assuming an … [View More]

3 5

Issue with Enforcing GPU Usage Limits in Slurm
by lyz＠simplehpc.com 16 Apr '25

16 Apr '25

Hi, I am currently encountering an issue with Slurm's GPU resource limitation. I have attempted to restrict the number of GPUs a user can utilize by executing the following command: sacctmgr modify user lyz set MaxTRES=gres/gpu=2 This command is intended to limit user 'lyz' to using a maximum of 2 GPUs. However, when the user submits a job using srun, specifying CUDA 0, 1, 2, and 3 in the job script, or os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3", the job still utilizes all 4 GPUs during … [View More]

4 14

pam_slurm_adopt and multiple jobs on the same worker node
by Massimo Sgaravatto 14 Apr '25

14 Apr '25

Dear all With the pam_slurm_adopt module as far as I understand you can ssh to a worker node if there is at least a job running on the node by that user. If there are multiple jobs, if I am not wrong you will be "mapped" to the last job started on the node. And, if you are using cgroups, you will be "confined" to the resources assigned to this "last" job. Is it possible in some way to specify the job to be mapped, in case there are multiple jobs for that user on the same node ? Thanks, Massimo

2 2

sllurmrestd via unix socket
by Brian Andrus 10 Apr '25

10 Apr '25

All, Maybe someone has seen this. I have slurmrestd running listening on port 8081 as well as a unix socket at /run/slurmrestd/slurmrestd.sock I am able to query the port with curl and do a ping. Everything seems fine. Other commands work as well. "pings": [ { "hostname": "head01", "pinged": "UP", "responding": true, "latency": 1605, &… [View More]

1 0

Minimum cpu cores per node partition level configuration
by Jeherul Islam 10 Apr '25

10 Apr '25

Dear All, I need to configure the slurm so the user must take a certain minimum number of CPU cores for a particular partition(not system-wide). Otherwise, the job must not run. Any suggestions will be highly appreciated. With Thanks and Regards -- Jeherul Islam

4 4

weird sacct behavior?
by Pierre Abele 10 Apr '25

10 Apr '25

Hi everyone, I am currently stuck with an sacct issue and would appreciate any help/hints/ideas: My users cannot retrieve job data from their currently running jobs through sacct anymore. Running sacct -a as root also reproduces this issue: It does not show running jobs, but both sacct -j <JobID> and squeue -j <JobID> do. AFAICT, this is not intended behavior (?). Also including longer time windows witch sacct -S ... -E did not help. root@slurmmaster:~# sacct -a | grep … [View More]

1 0

Re: slurmctld HA ; backup controller doesn't schedule and start any job
by Hiromasa Watanabe 10 Apr '25

10 Apr '25

Hi all, Fortunately I solved this problem by changing Slurm version from 22.05.9 to 23.11.10. With Slurm 23.11.10, after stopping the primary slurmctld and slurmdbd, when I submit a job with sbatch while backup slurmctld and slurmdbd are running, the job becomes scheduled and runs. I don't know why, but this is OK. Thanks, Hiro

1 0

errors while trying to setup slurmdbd.
by Steven Jones 09 Apr '25

09 Apr '25

root@vuwunicohpcdbp1 ~]# systemctl status slurmdbd × slurmdbd.service - Slurm DBD accounting daemon Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Thu 2025-04-10 10:28:52 NZST; 2min 33s ago Duration: 85ms Process: 2413 ExecStart=/usr/sbin/slurmdbd -D -s $SLURMDBD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 2413 (code=exited, status=1/FAILURE) CPU: 11ms Apr 10 10:28:52 … [View More]vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: Started Slurm DBD accounting daemon. Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: accounting_storage/as_mysql: _check_mysql_concat_is_sane: MySQL server version > Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: accounting_storage/as_mysql: init: Accounting storage MYSQL plugin loaded Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: fatal: This host not configured to run SlurmDBD ((vuwunicohpcdbp1 or vuwunicohp> Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: slurmdbd.service: Main process exited, code=exited, status=1/FAILURE Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: slurmdbd.service: Failed with result 'exit-code'. lines 1-14/14 (END) This is a current RHEL9.5 [root@vuwunicohpcdbp1 ~]# rpm -qi mariadb Name : mariadb Epoch : 3 Version : 10.5.27 Release : 1.el9_5 Architecture: x86_64 Install Date: Fri 04 Apr 2025 11:13:14 AM NZDT Group : Unspecified Size : 18784786 License : GPLv2 and LGPLv2 Signature : RSA/SHA256, Fri 06 Dec 2024 10:47:26 PM NZDT, Key ID 199e2f91fd431d51 Source RPM : mariadb-10.5.27-1.el9_5.src.rpm Build Date : Thu 05 Dec 2024 03:22:03 PM NZDT Build Host : x86-64-04.build.eng.rdu2.redhat.com Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : http://mariadb.org Summary : A very fast and robust SQL database server Description : MariaDB is a community developed fork from MySQL - a multi-user, multi-threaded SQL database server. It is a client/server implementation consisting of a server daemon (mariadbd) and many different client programs and libraries. The base package contains the standard MariaDB/MySQL client programs and utilities. [root@vuwunicohpcdbp1 ~]# ========= # Example slurmdbd.conf file. # # See the slurmdbd.conf man page for more information. # # Archive info #ArchiveJobs=yes #ArchiveDir="/tmp" #ArchiveSteps=yes #ArchiveScript= #JobPurge=12 #StepPurge=1 # # Authentication info AuthType=auth/munge #AuthInfo=/var/run/munge/munge.socket.2 # # slurmDBD info DbdAddr=127.0.0.1 #DbdAddr=localhost DbdHost=vuwunicoslurmrp1.ods.vuw.ac.nz #DbdHost=localhost DbdPort=6819 SlurmUser=slurm #MessageTimeout=300 DebugLevel=verbose #DefaultQOS=normal,standby LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurmdbd/slurmdbd.pid #PluginDir=/usr/lib/slurm #PrivateData=accounts,users,usage,jobs #TrackWCKey=yes # # Database info StorageType=accounting_storage/mysql #StorageHost=localhost StorageHost=127.0.0.1 #StoragePort=1234 StoragePort=3306 #StoragePort=6819 #StoragePass=xxxxx StoragePass=xxxxxx #StoragePass=aPIkaEqxLrA StorageUser=slurm StorageLoc=slurm_acct_db #ssj # ##ssj fini ============ This ran in test just fine but the slurm version is slightly newer in this case, [root@vuwunicohpcdbp1 admjonesst1]# rpm -qa |grep slurm slurm-24.11.3-1.el9.x86_64 slurm-slurmdbd-24.11.3-1.el9.x86_64 [root@vuwunicohpcdbp1 admjonesst1]# rpm -qi slurm-slurmdbd-24.11.3-1.el9.x86_64 Name : slurm-slurmdbd Version : 24.11.3 Release : 1.el9 Architecture: x86_64 Install Date: Mon 07 Apr 2025 01:58:32 PM NZST Group : System Environment/Base Size : 2959288 License : GPLv2+ Signature : (none) Source RPM : slurm-24.11.3-1.el9.src.rpm Build Date : Mon 07 Apr 2025 12:54:21 PM NZST Build Host : vuwunicoslurmp1.ods.vuw.ac.nz URL : https://slurm.schedmd.com/ Summary : Slurm database daemon Description : Slurm database daemon. Used to accept and process database RPCs and upload database changes to slurmctld daemons on each cluster [root@vuwunicohpcdbp1 admjonesst1]# regards Steven [View Less]

2 2

pam error, related to accounting?
by David Bremner 09 Apr '25

09 Apr '25

Recently I enabled accounting on my tiny (1 compute node, one head node) slurm cluster. slurmdbd.conf looks like AuthType=auth/munge DbdHost=vertex DbdPort=6819 SlurmUser=slurm StorageHost=localhost StorageType=accounting_storage/mysql StorageUser=slurm StoragePass=#elided# StoragePort=3306 StorageLoc=slurm_acct_db PurgeEventAfter=30days PurgeJobAfter=30days PurgeResvAfter=30days PurgeStepAfter=30days PurgeSuspendAfter=30days Some … [View More]

1 0

recommended freeIPMI version
by Heckes, Frank 09 Apr '25

09 Apr '25

Hi all, I’d like to update to SLURM version 24.11.4. I was searching for a recommendation for freeIPMI (power measurement), but couldn’t find one on the schedMD web-pages (sorry, if I overlooked it). The latest available freeIPMI is 1.6.15 (https://ftp.gnu.org/gnu/freeipmi/freeipmi-1.6.15.tar.gz). Does anyone knows whether this is the best choice? Many thanks in advance for any advice. Cheers, -Frank

3 4

slurmctld HA ; backup controller doesn't schedule and start any job
by hiromasa.watanabe＠gmail.com 09 Apr '25

09 Apr '25

slurmctld HA ; backup controller doesn't schedule and start any job Hi all, I am trying a slurmctld HA configuration on two servers, using slurm version 22.05.9 of AlmaLinux 9.4. The problem is, after stopping the primary slurmctld and slurmdbd, when I submit a job with sbatch while backup slurmctld and slurmdbd are running, the job will be pending with Reason=None, and it will not be scheduled and will not start running. $ squeue JOBID PARTITION NAME USER ST TIME … [View More]NODES NODELIST(REASON) 43 cpu_multi job hpc PD 0:00 1 (None) Why won't my job start running? What should I change to get the job to start running? The slurmctld.log and configuration files are shown below. Backup's slurmctld.log： [2025-04-09T15:31:17.000] debug3: Heartbeat at 1744180276 [2025-04-09T15:31:18.000] debug3: Heartbeat at 1744180277 [2025-04-09T15:31:19.022] debug3: Heartbeat at 1744180279 [2025-04-09T15:31:19.605] debug2: Processing RPC: REQUEST_SUBMIT_BATCH_JOB from UID=1000 [2025-04-09T15:31:19.605] debug3: _set_hostname: Using auth hostname for alloc_node: compute1 [2025-04-09T15:31:19.605] debug3: JobDesc: user_id=1000 JobId=N/A partition=cpu_multi name=job [2025-04-09T15:31:19.605] debug3: cpus=2-4294967294 pn_min_cpus=1 core_spec=-1 [2025-04-09T15:31:19.605] debug3: Nodes=1-[1] Sock/Node=65534 Core/Sock=65534 Thread/Core=65534 [2025-04-09T15:31:19.605] debug3: pn_min_memory_job=18446744073709551615 pn_min_tmp_disk=-1 [2025-04-09T15:31:19.605] debug3: immediate=0 reservation=(null) [2025-04-09T15:31:19.605] debug3: features=(null) batch_features=(null) cluster_features=(null) prefer=(null) [2025-04-09T15:31:19.605] debug3: req_nodes=(null) exc_nodes=(null) [2025-04-09T15:31:19.605] debug3: time_limit=-1--1 priority=-1 contiguous=0 shared=-1 [2025-04-09T15:31:19.605] debug3: kill_on_node_fail=-1 script=#! /bin/bash #SBATCH -p cpu_multi #SBATC... [2025-04-09T15:31:19.605] debug3: argv="./twocore.sh" [2025-04-09T15:31:19.605] debug3: environment=SHELL=/bin/bash,PYENV_SHELL=bash,HISTCONTROL=ignoredups,... [2025-04-09T15:31:19.605] debug3: stdin=/dev/null stdout=/misc/home/hpc/slurmtest/twocore_%J.out stderr=(null) [2025-04-09T15:31:19.605] debug3: work_dir=/misc/home/hpc/slurmtest alloc_node:sid=compute1:281600 [2025-04-09T15:31:19.605] debug3: power_flags= [2025-04-09T15:31:19.605] debug3: resp_host=(null) alloc_resp_port=0 other_port=0 [2025-04-09T15:31:19.605] debug3: dependency=(null) account=(null) qos=(null) comment=(null) [2025-04-09T15:31:19.605] debug3: mail_type=0 mail_user=(null) nice=0 num_tasks=2 open_mode=0 overcommit=-1 acctg_freq=(null) [2025-04-09T15:31:19.605] debug3: network=(null) begin=Unknown cpus_per_task=1 requeue=-1 licenses=(null) [2025-04-09T15:31:19.605] debug3: end_time= signal=0@0 wait_all_nodes=-1 cpu_freq= [2025-04-09T15:31:19.605] debug3: ntasks_per_node=-1 ntasks_per_socket=-1 ntasks_per_core=-1 ntasks_per_tres=-1 [2025-04-09T15:31:19.605] debug3: mem_bind=0:(null) plane_size:65534 [2025-04-09T15:31:19.605] debug3: array_inx=(null) [2025-04-09T15:31:19.605] debug3: burst_buffer=(null) [2025-04-09T15:31:19.605] debug3: mcs_label=(null) [2025-04-09T15:31:19.605] debug3: deadline=Unknown [2025-04-09T15:31:19.605] debug3: bitflags=0x1a00c000 delay_boot=4294967294 [2025-04-09T15:31:19.605] debug3: assoc_mgr_fill_in_user: found correct user: hpc(1000) [2025-04-09T15:31:19.605] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=hpc(1000), acct=hpc, cluster=cluster, partition=cpu_multi [2025-04-09T15:31:19.605] debug3: assoc_mgr_fill_in_assoc: found correct association of user=hpc(1000), acct=hpc, cluster=cluster, partition=cpu_multi to assoc=16 acct=hpc [2025-04-09T15:31:19.605] debug3: found correct qos [2025-04-09T15:31:19.607] debug2: priority/multifactor: priority_p_set: initial priority for job 44 is 33 [2025-04-09T15:31:19.607] debug2: found 1 usable nodes from config containing compute1 [2025-04-09T15:31:19.607] debug2: found 1 usable nodes from config containing compute2 [2025-04-09T15:31:19.607] debug3: _pick_best_nodes: JobId=44 idle_nodes 2 share_nodes 2 [2025-04-09T15:31:19.607] debug2: select/cons_tres: select_p_job_test: evaluating JobId=44 [2025-04-09T15:31:19.607] debug2: sched: JobId=44 allocated resources: NodeList=(null) [2025-04-09T15:31:19.607] _slurm_rpc_submit_batch_job: JobId=44 InitPrio=33 usec=2490 [2025-04-09T15:31:19.608] debug3: create_mmap_buf: loaded file `/var/spool/slurm/ctld/job_state` as buf_t [2025-04-09T15:31:19.609] debug3: Writing job id 45 to header record of job_state file [2025-04-09T15:31:21.000] debug3: Heartbeat at 1744180280 [2025-04-09T15:31:21.257] debug2: _slurm_connect: failed to connect to 192.168.56.11:6817: Connection refused [2025-04-09T15:31:21.257] debug2: Error connecting slurm stream socket at 192.168.56.11:6817: Connection refused [2025-04-09T15:31:22.000] debug3: Heartbeat at 1744180282 [2025-04-09T15:31:24.000] debug3: Heartbeat at 1744180283 [2025-04-09T15:31:25.004] debug3: Heartbeat at 1744180285 [2025-04-09T15:31:27.001] debug3: Heartbeat at 1744180287 [2025-04-09T15:31:29.000] debug3: Heartbeat at 1744180288 [2025-04-09T15:31:30.000] debug3: Heartbeat at 1744180289 [2025-04-09T15:31:31.006] debug3: Heartbeat at 1744180291 [2025-04-09T15:31:32.822] debug2: _slurm_connect: failed to connect to 192.168.56.11:6817: Connection refused [2025-04-09T15:31:32.822] debug2: Error connecting slurm stream socket at 192.168.56.11:6817: Connection refused [2025-04-09T15:31:33.007] debug3: Heartbeat at 1744180293 [2025-04-09T15:31:35.000] debug3: Heartbeat at 1744180294 [2025-04-09T15:31:36.002] debug3: Heartbeat at 1744180296 [2025-04-09T15:31:36.395] debug2: select/cons_tres: select_p_job_test: evaluating JobId=43 [2025-04-09T15:31:36.395] debug2: select/cons_tres: select_p_job_test: evaluating JobId=44 [2025-04-09T15:31:38.000] debug3: Heartbeat at 1744180297 [2025-04-09T15:31:38.497] debug2: Performing purge of old job records [2025-04-09T15:31:39.000] debug3: Heartbeat at 1744180298 [2025-04-09T15:31:40.000] debug3: Heartbeat at 1744180300 [2025-04-09T15:31:40.655] debug2: Testing job time limits and checkpoints [2025-04-09T15:31:42.000] debug3: Heartbeat at 1744180301 [2025-04-09T15:31:43.000] debug3: Heartbeat at 1744180302 slurm.conf: ClusterName=cluster SlurmctldHost=gateway1 #Primary(192.168.56.11) SlurmctldHost=gateway2 #Backup(192.168.56.12) MpiDefault=pmix ProctrackType=proctrack/cgroup PrologFlags=Contain ReturnToService=0 SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurm/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurm/d SlurmUser=slurm StateSaveLocation=/var/spool/slurm/ctld SwitchType=switch/none TaskEpilog=/etc/slurm/taskepilog.sh TaskPlugin=task/cgroup,task/affinity TaskProlog=/etc/slurm/taskprolog.sh InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=10 SlurmdTimeout=300 Waittime=0 DefMemPerCPU=32 SchedulerType=sched/builtin SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory PriorityType=priority/multifactor PriorityWeightPartition=1000 AccountingStorageHost=gateway1 AccountingStorageBackupHost=gateway2 AccountingStorageType=accounting_storage/slurmdbd AccountingStoreFlags=job_comment JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=debug5 SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log NodeName=compute1 CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=3900 Weight=1 NodeName=compute2 CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=3900 Weight=1 PartitionName=gpu_single Nodes=ALL PriorityJobFactor=30 MaxTime=INFINITE State=UP Default=YES PartitionName=cpu_single Nodes=ALL PriorityJobFactor=10 MaxTime=INFINITE State=UP PartitionName=cpu_multi Nodes=ALL MaxTime=INFINITE State=UP slurmdbd.conf: AuthType=auth/munge DebugLevel=4 DbdHost=gateway1 DbdBackupHost=gateway2 LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurm/slurmdbd.pid PurgeEventAfter=1month PurgeJobAfter=1month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month PurgeTXNAfter=1month PurgeUsageAfter=1month SlurmUser=slurm StorageType=accounting_storage/mysql StorageHost=gateway1 StoragePass=mypassword StorageUser=slurm Best regards, Hiro [View Less]

1 0

Slurm version 24.11.4 is now available
by Marshall Garey 08 Apr '25

08 Apr '25

We are pleased to announce the availability of Slurm version 24.11.4. This release fixes a variety of major to minor severity bugs. Some edge cases that caused jobs to pend forever are fixed. Notable stability issues that are fixed include: * slurmctld crashing upon receiving a certain heterogeneous job submission. * slurmd crashing after a communications failure with a slurmstepd. * A variety of race conditions related to receiving and processing connections, including one that resulted in … [View More]the slurmd ignoring new RPC connections. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > -- slurmctld,slurmrestd - Avoid possible race condition that could have caused > process to crash when listener socket was closed while accepting a new > connection. > -- slurmrestd - Avoid race condition that could have resulted in address > logged for a UNIX socket to be incorrect. > -- slurmrestd - Fix parameters in OpenAPI specification for the following > endpoints to have "job_id" field: > GET /slurm/v0.0.40/jobs/state/ > GET /slurm/v0.0.41/jobs/state/ > GET /slurm/v0.0.42/jobs/state/ > GET /slurm/v0.0.43/jobs/state/ > -- slurmd - Fix tracking of thread counts that could cause incoming > connections to be ignored after burst of simultaneous incoming connections > that trigger delayed response logic. > -- Stepmgr - Avoid unnecessary SRUN_TIMEOUT forwarding to stepmgr. > -- Fix jobs being scheduled on higher weighted powered down nodes. > -- Fix how backfill scheduler filters nodes from the available nodes based on > exclusive user and mcs_label requirements. > -- acct_gather_energy/{gpu,ipmi} - Fix potential energy consumption adjustment > calculation underflow. > -- acct_gather_energy/ipmi - Fix regression introduced in 24.05.5 (which > introduced the new way of preserving energy measurements through slurmd > restarts) when EnergyIPMICalcAdjustment=yes. > -- Prevent slurmctld deadlock in the assoc mgr. > -- Fix memory leak when RestrictedCoresPerGPU is enabled. > -- Fix preemptor jobs not entering execution due to wrong calculation of > accounting policy limits. > -- Fix certain job requests that were incorrectly denied with node > configuration unavailable error. > -- slurmd - Avoid crash due when slurmd has a communications failure with > slurmstepd. > -- Fix memory leak when parsing yaml input. > -- Prevent slurmctld from showing error message about PreemptMode=GANG being a > cluster-wide option for `scontrol update part` calls that don't attempt to > modify partition PreemptMode. > -- Fix setting GANG preemption on partition when updating PreemptMode with > scontrol. > -- Fix CoreSpec and MemSpec limits not being removed from previously > configured slurmd. > -- Avoid race condition that could lead to a deadlock when slurmd, slurmstepd, > slurmctld, slurmrestd or sackd have a fatal event. > -- Fix jobs using --ntasks-per-node and --mem keep pending forever when the > requested mem divided by the number of cpus will surpass the configured > MaxMemPerCPU. > -- slurmd - Fix address logged upon new incoming RPC connection from "INVALID" > to IP address. > -- Fix memory leak when retrieving reservations. This affects scontrol, sinfo, > sview, and the following slurmrestd endpoints: > 'GET /slurm/{any_data_parser}/reservation/{reservation_name}' > 'GET /slurm/{any_data_parser}/reservations' > -- Log warning instead of debuflags=conmgr gated log when deferring new > incoming connections when number of active connections exceed > conmgr_max_connections. > -- Avoid race condition that could result in worker thread pool not activating > all threads at once after a reconfigure resulting in lower utilization of > available CPU threads until enough internal activity wakes up all threads > in the worker pool. > -- Avoid theoretical race condition that could result in new incoming RPC > socket connections being ignored after reconfigure. > -- slurmd - Avoid race condition that could result in a state where new > incoming RPC connections will always be ignored. > -- Add ReconfigFlags=KeepNodeStateFuture to restore saved FUTURE node state on > restart and reconfig instead of reverting to FUTURE state. This will be > made the default in 25.05. > -- Fix case where hetjob submit would cause slurmctld to crash. > -- Fix jobs using --cpus-per-gpu and --mem keep pending forever when the > requested mem divided by the number of cpus will surpass the configured > MaxMemPerCPU. > -- Enforce that jobs using --mem and several --*-per-* options do not violate > the MaxMemPerCPU in place. > -- slurmctld - Fix use-cases of jobs incorrectly pending held when --prefer > features are not initially satisfied. > -- slurmctld - Fix jobs incorrectly held when --prefer not satisfied in some > use-cases. > -- Ensure RestrictedCoresPerGPU and CoreSpecCount don't overlap. [View Less]

1 0

slurm send email status with no details
by Oren 07 Apr '25

07 Apr '25

Hi, We set up a slurm system with email notification, this is the slurm.conf `MailProg=/usr/sbin/sendmail` But the email that I get has not status, just an empty message: [image: image.png] no subject, no info, what are we missing? Thanks~

2 1

Run a command in Slurm with all streams and signals connected to the submitting command
by Michael Milton 06 Apr '25

06 Apr '25

I'm helping with a workflow manager that needs to submit Slurm jobs. For logging and management reasons, the job (e.g. srun python) needs to be run as though it were a regular subprocess (python): - stdin, stdout and stderr for the command should be connected to process inside the job - signals sent to the command should be sent to the job process - We don't want to use the existing job allocation, if this is run from a Slurm job - The command should only terminate when the … [View More]

2 2

Re: Run a command in Slurm with all streams and signals connected to the submitting command
by Michael Milton 04 Apr '25

04 Apr '25

Thanks Davide, It's true that srun will create an allocation if you aren't inside a job, but if you are inside a job and you request more resources than it has, then srun will just fail. This is the key issue that I want to avoid. On Sat, Apr 5, 2025 at 11:48 AM Davide DelVento <davide.quantum(a)gmail.com> wrote: > The plain srun is probably the best bet, and if you really need the thing > to be started from another slurm job (rather than the login node) you will > need to … [View More]

1 0

Best Way to See GPUs in Use?
by Jason Simms 02 Apr '25

02 Apr '25

Hello all, Apologies for the basic question, but is there a straightforward, best-accepted method for using Slurm to report on which GPUs are currently in use? I've done some searching and people recommend all sorts of methods, including parsing the output of nvidia-smi (seems inefficient, especially across multiple GPU nodes), as well as using other tools such as Grafana, XDMoD, etc. We do track GPUs as a resource, so I'd expect I could get at the info with sreport or something like that, … [View More]

3 2

slum job sumisison using different UID/GID
by navin srivastava 02 Apr '25

02 Apr '25

Hi, Can munge depend upon the UID/GID for the job submission? There is a scenario where the cluster is running with the NIS authentication mechanism. There is a login Node created which is part of LDAP and we installed the slurm to interact with slurm master. I am able to run jobs using root but for users it is not working as the UID/GID differ. Is there a way we can overcome this issue? and be able to run jobs. Rehards Navin

1 0

slurm releases
by Patrick Begou 01 Apr '25

01 Apr '25

Hi slurm team, I would ask some clarifications with slurm releases. Why two versions of slurm are available ? I speak of 24.05.7 versus 24.11.3 on https://www.schedmd.com/slurm-support/release-announcements and announces made on this list ? I'm managing small clusters in a french public research lab and I've successfully updated (lets says re-installed as it was very old) an old one with slurm-24.05.3 some month ago. Now I'm trying to deploy a second cluster with the same approach … [View More]

2 1

cpus and gpus partitions and how to optimize the resource usage
by Massimo Sgaravatto 31 Mar '25

31 Mar '25

Dear all We have just installed a small SLURM cluster composed of 12 nodes: - 6 CPU only nodes: 2 Sockets=2, 96 CoresPerSocket 2, ThreadsPerCore=2, 1.5 TB of RAM - 6 nodes with also GPUS: same conf of the CPU-only node + 4 H100 per node We started with a setup with 2 partitions: - a 'onlycpus' partition which sees all the cpu-only nodes - a 'gpus' partition which sees the nodes with gpus and asked users to use the 'gpus' partition only for jobs that need gpus (for the time being we are … [View More]

4 5

Preemption question
by Kamil Wilczek 31 Mar '25

31 Mar '25

Dear All, I would like to be able to preempt (SUSPEND) a single QoS of a user that blocks the queue for several days. Currently I have about 100 users on the cluster and it seems that setting the "Preempt" option to each QoS (we have personal QoSes) is not optimal. https://slurm.schedmd.com/sacctmgr.html#OPT_Preempt Is there a way to to set an option to this single problematic QoS, saying that the QoS can be preempted by any other QoS? It would be much more administrator-friendly solution … [View More]

3 4

Slurm 24.05 and OpenMPI
by Matthias Leopold 28 Mar '25

28 Mar '25

Hi, I built a small Slurm 21.08 cluster with NVIDIA GPU hardware and NVIDIA deepops framework a couple of years ago. It is based on Ubuntu 20.04 and makes use of the NVIDIA pyxis/enroot container solution. For operational validation I used the nccl-tests application in a container. nccl-tests is compiled with MPI support (OpenMPI 4.1.6 or 4.1.7) and I used it also for validation of MPI jobs. Slurm jobs use "pmix" and tasks are launched via srun (not mpirun). Some of the GPUs can talk to … [View More]

3 10

Doubt with SelectTypeParameters in slurm.conf
by Gestió Servidors 28 Mar '25

28 Mar '25

Hello, I'm running some tests in a very small testing environment (before applying in the real scenario). My environment is only a computer with a old Intel i4 with this "lscpu" configuration: Architecture: x86_64 CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 26 Model name: … [View More]

2 1

bit_cache_init failure on the second time backup controller tries to take control
by Safdar Iqbal 27 Mar '25

27 Mar '25

Hi, We're running into an issue where slurmctld core-dumps with the following error. This happens on the backup controller, if it needs to take over from the primary, _for a second time_. slurmctld: fatal: bit_cache_init: cannot change size once set Has anyone seen this error before? Also if there are any existing discussions and/or tickets related to this, please let me know. Our slurm version is 24.11.1. ________________ Steps to reproduce: 1. On a healthy cluster, we make the primary … [View More]

1 0

How limit CPUs per node in a partition
by Gestió Servidors 27 Mar '25

27 Mar '25

Hello, I have a testing partition with only a node. That server has 12 CPUs (it's a very old server) (2 sockets, 6 cores per socket, 1 thread per core). That partition, called "test.q" only has that node, so by default, partition test.q has 12 CPUs (all from testing node). However, now I would like to decrease that CPU number from 12 to 4, for example. "slurmd -C" returns "CPUs=12 Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1" as I have said to you before and if in slurm.conf I … [View More]

1 0

Re: Using more cores/CPUs that requested with
by Shunran Zhang 26 Mar '25

26 Mar '25

Ugh I think I did not catch up with the docs. I started with a system that defaults to cgroup v1 but the Slurm doc for that plugin is NOT available at that time. Thus I converted everything to cgroup v2. It appears that they are both supported and that documentation issue is more on the dev side than admin side. Thanks for pointing that out. I misinterpreted the "coming soon" part of cgroup v1 plugin and the "legacy" naming for "do not use". It should be fine. 2025年3月27日(木) 0:48 Williams, … [View More]

1 0

Re: Using more cores/CPUs that requested with
by Gestió Servidors 26 Mar '25

26 Mar '25

Hello, Thanks for your answers. I will try now!! One more question: is there any way to check if Cgroups restrictions is working fine during a "running" job or during SLURM scheduling process? Thanks again!

4 3

Using more cores/CPUs that requested with sbatch
by Gestió Servidors 25 Mar '25

25 Mar '25

Hello, I would like to know if there is any mechanism to avoid a user to do "cheating" when he submits a job. For example, a user submits a OpenMP job requesting 1 node (-N 1) and 2 tasks (-n 2). However, inside his script (or compiled in his binany), user uses more than 2 ntasks, for example, all free cores. For SLURM, job is only using 2, but really, node is 100% assigned to that job. SLURM can control that "abuse" in some way? Cgroups? Thanks.

3 2

Re: Using more cores/CPUs that requested with sbatch
by Megan 25 Mar '25

25 Mar '25

Slurm can restrict jobs to their allocated resources using cgroups. To use cgroups you will have to configure the TaskPlugin cgroup option and specify how the cgroups should behave in cgroup.conf. slurm.conf TaskPlugin=task/cgroup Slurm Workload Manager - slurm.conf <https://slurm.schedmd.com/slurm.conf.html#OPT_task/cgroup> cgroup.conf ConstrainCores=yes Slurm Workload Manager - cgroup.conf <https://slurm.schedmd.com/cgroup.conf.html#OPT_ConstrainCores> Regards, --Megan

1 0

slurmrestd equivalent to "srun -n 10 echo HELLO"
by Dan Healy 24 Mar '25

24 Mar '25

Hi Slurm Community, I'm starting to experiment with slurmrestd for a new app we're writing. I'm having trouble understanding one aspect of submitting jobs. When I run something like `srun -n 10 echo HELLO', I get HELLO returned to my console/stdout 10x. When I submit this command as a script to the /jobs/submit route, I get success/200, but *I cannot determine how to get the console output of HELLO 10x in any form*. It's not in my stdout log for that job even though I can verify that the job … [View More]

3 2

2025

2024

slurm-users ----- 2025 ----- May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users