- slurm-users - lists.schedmd.com

Slurm release candidate version 25.05.0rc1 is available for testing
by Marshall Garey 13 May '25

13 May '25

We are pleased to announce the availability of Slurm release candidate 25.05.0rc1. To highlight some new features coming in 25.05: - Support for defining multiple topology configurations, and varying them by partition. - Support for tracking and allocating hierarchical resources. - Dynamic nodes can be dynamically added to the topology. - topology/block - Allow for gaps in the block layout. - Support for encrypting all network communication with TLS. - jobcomp/kafka - Optionally send job info at job start as well as job end. - Support an OR operator in --license requests. - switch/hpe_slingshot - Support for > 252 ranks per node. - switch/hpe_slingshot - Support mTLS authentication to the fabric manager. - sacctmgr - Add support for dumping and loading QOSes. - srun - Add new --wait-for-children option to keep the step running until all launched processes have been launched (cgroup/v2 only). - slurmrestd - Add new endpoint for creating reservations. This is the first release candidate of the upcoming 25.05 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 25.05.x version and we will address them before the first production 25.05.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/download-slurm/. The changelog for 25.05.0rc1 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-25.05.md#chang… -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Performance Issues after Update to 24.11.5
by Tilman Hoffbauer 13 May '25

13 May '25

Hello, we are running a SLURM-managed cluster with one control node (g-vm03) and 26 worker nodes (ouga[03-28]) on Rocky 8. We recently updated from 20.11.9 through 23.02.8 to 24.11.0 and then 24.11.5. Since then, we are experiencing performance issues - squeue and scontrol ping are slow to react and sometimes deliver "timeout on send/recv" messages, even with only very few parallel requests. We did not experience these issues with SLURM 20.11.9 before, we did not check the intermediate version 23.02.8 in detail before. In the log of slurmctld, we can also find messages like slurmctld: error: slurm_send_node_msg: [socket:[1272743]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error We thus implemented all recommendations from the high throughput documentation, and did achieve improvements with it (most notably by increasing the maximum number of open files and increasing MessageTimeout and TCPTimeout). For debugging, I attached the slurm.conf, the sdiag output (the server thread count is almost always 1 and sometimes increases to 2), the slurmctld log and the slurmdbd log from a time of high load. We would be very thankful for any input on how restore the old performance. Kind Regards, Tilman Hoffbauer

2 4

Do I have to hold back RAM for worker nodes?
by Xaver Stiensmeier 13 May '25

13 May '25

Dear Slurm-User List, currently, in our slurm.conf, we are setting: SelectType: select/cons_tres SelectTypeParameters: CR_Core and in our node configuration /RealMemory /was basically reduced by an amount to make sure the node always had enough RAM to run the OS. However, this is apparently now how it is supposed to be done: Lowering RealMemory with the goal of setting aside some amount for the OS and not available for job allocations will not work as intended if Memory is not set as a consumable resource in *SelectTypeParameters*. So one of the *_Memory options need to be enabled for that goal to be accomplished. (https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory) This leads to four questions regarding holding back RAM for worker nodes. Answers/help with any of those questions would be appreciated. *1.* Is reserving enough RAM for the worker node's OS and slurmd actually a thing you have to manage? *2.* If so how can we reserve enough RAM for the worker node's OS and slurmd when using CR_Core? *3.* Is that maybe a strong argument against using CR_Core that we overlooked? And semi-related: https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory talks about taking a value in megabytes. *4.* Is RealMemory really expecting megabytes or is it mebibytes? Best regards, Xaver

6 8

slurm failing to start as it doesnt know its node name??????
by Steven Jones 12 May '25

12 May '25

Hi, I am not sure what i have missed but I am getting this error on a compute node. ====== root@vm01no16 ~]# systemctl status slurmd.service × slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Sun 2025-05-11 23:01:29 UTC; 15min ago Process: 1178 ExecStart=/usr/sbin/slurmd --systemd $SLURMD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 1178 (code=exited, status=1/FAILURE) CPU: 3ms May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: Starting Slurm node daemon... May 11 23:01:29 vm01no16.ods.vuw.ac.nz slurmd[1178]: slurmd: fatal: Unable to determine this slurmd's NodeName May 11 23:01:29 vm01no16.ods.vuw.ac.nz slurmd[1178]: fatal: Unable to determine this slurmd's NodeName May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: slurmd.service: Main process exited, code=exited, status=1/FAIL> May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: slurmd.service: Failed with result 'exit-code'. May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: Failed to start Slurm node daemon. [root@vm01no16 ~]# uname -a Linux vm01no16.ods.vuw.ac.nz 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 30 17:38:54 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux [root@vm01no16 ~]# vi /etc/hosts [root@vm01no16 ~]# cat /etc/hostname vm01no16.ods.vuw.ac.nz [root@vm01no16 ~]# tail /etc/hosts # Entry for int01no9.ods.vuw.ac.nz 130.195.87.29 int01no9.ods.vuw.ac.nz int01no9.ods.vuw.ac.nz-default # Entry for vm01no13.ods.vuw.ac.nz 130.195.87.41 vm01no13.ods.vuw.ac.nz vm01no13.ods.vuw.ac.nz-default # Entry for vm01no14.ods.vuw.ac.nz 130.195.87.42 vm01no14.ods.vuw.ac.nz vm01no14.ods.vuw.ac.nz-default # Entry for vm01no15.ods.vuw.ac.nz 130.195.87.43 vm01no15.ods.vuw.ac.nz vm01no15.ods.vuw.ac.nz-default # Entry for vm01no16.ods.vuw.ac.nz 130.195.87.44 vm01no16.ods.vuw.ac.nz vm01no16.ods.vuw.ac.nz-default [root@vm01no16 ~]# ====== regards Steven Jones B.Eng (Hons) Technical Specialist - Linux RHCE Victoria University, Digital Solutions, Level 8 Rankin Brown Building, Wellington, NZ 6012 0064 4 463 6272

2 1

Slurm versions 24.11.5, 24.05.8, and 23.11.11 are now available (CVE-2025-43904)
by Tim Wickberg 07 May '25

07 May '25

Slurm versions 24.11.5, 24.05.8, and 23.11.11 are now available and include a fix for a recently discovered security issue. SchedMD customers were informed on April 23rd and provided a patch on request; this process is documented in our security policy. [1] A mistake with permission handling for Coordinators within Slurm's accounting system can allow a Coordinator to promote a user to Administrator. (CVE-2025-43904) Thank you to Sekou Diakite (HPE) for reporting this. Downloads are available at https://www.schedmd.com/downloads.php . Release notes follow below. - Tim [1] https://www.schedmd.com/security-policy/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support > * Changes in Slurm 24.11.5 > ========================== > -- Return error to scontrol reboot on bad nodelists. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.40 > endpoints. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.41 > endpoints. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.42 > endpoints. > -- data_parser/v0.0.42 - Added +inline_enums flag which modifies the > output when generating OpenAPI specification. It causes enum arrays to not > be defined in their own schema with references ($ref) to them. Instead they > will be dumped inline. > -- Fix binding error with tres-bind map/mask on partial node allocations. > -- Fix stepmgr enabled steps being able to request features. > -- Reject step creation if requested feature is not available in job. > -- slurmd - Restrict listening for new incoming RPC requests further into > startup. > -- slurmd - Avoid auth/slurm related hangs of CLI commands during startup > and shutdown. > -- slurmctld - Restrict processing new incoming RPC requests further into > startup. Stop processing requests sooner during shutdown. > -- slurmcltd - Avoid auth/slurm related hangs of CLI commands during > startup and shutdown. > -- slurmctld: Avoid race condition during shutdown or reconfigure that > could result in a crash due delayed processing of a connection while > plugins are unloaded. > -- Fix small memleak when getting the job list from the database. > -- Fix incorrect printing of % escape characters when printing stdio > fields for jobs. > -- Fix padding parsing when printing stdio fields for jobs. > -- Fix printing %A array job id when expanding patterns. > -- Fix reservations causing jobs to be held for Bad Constraints > -- switch/hpe_slingshot - Prevent potential segfault on failed curl > request to the fabric manager. > -- Fix printing incorrect array job id when expanding stdio file names. > The %A will now be substituted by the correct value. > -- Fix printing incorrect array job id when expanding stdio file names. > The %A will now be substituted by the correct value. > -- switch/hpe_slingshot - Fix vni range not updating on slurmctld restart > or reconfigre. > -- Fix steps not being created when using certain combinations of -c and > -n inferior to the jobs requested resources, when using stepmgr and nodes > are configured with CPUs == Sockets*CoresPerSocket. > -- Permit configuring the number of retry attempts to destroy CXI service > via the new destroy_retries SwitchParameter. > -- Do not reset memory.high and memory.swap.max in slurmd startup or > reconfigure as we are never really touching this in slurmd. > -- Fix reconfigure failure of slurmd when it has been started manually and > the CoreSpecLimits have been removed from slurm.conf. > -- Set or reset CoreSpec limits when slurmd is reconfigured and it was > started with systemd. > -- switch/hpe-slingshot - Make sure the slurmctld can free step VNIs after > the controller restarts or reconfigures while the job is running. > -- Fix backup slurmctld failure on 2nd takeover. > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904. > * Changes in Slurm 24.05.8 > ========================== > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904. > * Changes in Slurm 23.11.11 > =========================== > -- Fixed a job requeuing issue that merged job entries into the same SLUID > when all nodes in a job failed simultaneously. > -- Add ABORT_ON_FATAL environment variable to capture a backtrace from any > fatal() message. > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904.

1 0

Lots of RPC calls and REQUEST_GETPW calls
by Guillette, Jeremy 07 May '25

07 May '25

Hello, I’m trying to figure out why we’ve been seeing an increase in network traffic in our AWS-based cluster, which uses Amazon’s parallel cluster tool. After an incident a couple weeks ago, I turned on debug2 logging on the slurmd processes, and I’m seeing huge numbers of `REQUEST_GETPW` and `REQUEST_GETGR` requests going to the slurmd processes. I briefly turned on debug2 logging for `slurmctld` as well, and I’m seeing lots of RPC calls, but not as many as the `REQUEST_GETPW` requests that I’ve seen on compute node slurmd processes. Here’s a sample from the slurmctld log: ``` [2025-04-28T15:11:05.436] debug2: _slurm_rpc_dump_partitions, size=1253 usec=20 [2025-04-28T15:11:05.450] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2971 [2025-04-28T15:11:05.451] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2971 [2025-04-28T15:11:05.451] debug2: _slurm_rpc_dump_partitions, size=1253 usec=16 [2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2788 [2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2788 [2025-04-28T15:11:05.461] debug2: _slurm_rpc_dump_partitions, size=1253 usec=12 [2025-04-28T15:11:05.517] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2916 [2025-04-28T15:11:05.518] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2916 [2025-04-28T15:11:05.518] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.628] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3405 [2025-04-28T15:11:05.629] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3405 [2025-04-28T15:11:05.629] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2189 [2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2189 [2025-04-28T15:11:05.740] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:05.845] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2209 [2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2209 [2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=4106 [2025-04-28T15:11:05.846] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.847] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=4106 [2025-04-28T15:11:05.847] debug2: _slurm_rpc_dump_partitions, size=1253 usec=11 [2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3400 [2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3400 [2025-04-28T15:11:05.938] debug2: _slurm_rpc_dump_partitions, size=1253 usec=18 [2025-04-28T15:11:06.903] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3449 [2025-04-28T15:11:06.904] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3449 [2025-04-28T15:11:06.904] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.175] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3722 [2025-04-28T15:11:07.176] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3722 [2025-04-28T15:11:07.177] debug2: _slurm_rpc_dump_partitions, size=1253 usec=254 [2025-04-28T15:11:07.205] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=4040 [2025-04-28T15:11:07.206] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=4040 [2025-04-28T15:11:07.206] debug2: _slurm_rpc_dump_partitions, size=1253 usec=17 [2025-04-28T15:11:07.237] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2990 [2025-04-28T15:11:07.238] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2990 [2025-04-28T15:11:07.239] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.284] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2920 [2025-04-28T15:11:07.285] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2920 [2025-04-28T15:11:07.285] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.370] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3236 [2025-04-28T15:11:07.371] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3236 [2025-04-28T15:11:07.371] debug2: _slurm_rpc_dump_partitions, size=1253 usec=17 [2025-04-28T15:11:08.463] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2848 [2025-04-28T15:11:08.464] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2848 [2025-04-28T15:11:08.464] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:08.691] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2627 [2025-04-28T15:11:08.692] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2627 [2025-04-28T15:11:08.692] debug2: _slurm_rpc_dump_partitions, size=1253 usec=18 [2025-04-28T15:11:08.873] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3729 [2025-04-28T15:11:08.874] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3729 [2025-04-28T15:11:08.875] debug2: _slurm_rpc_dump_partitions, size=1253 usec=196 [2025-04-28T15:11:08.881] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3461 [2025-04-28T15:11:08.882] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3461 [2025-04-28T15:11:08.882] debug2: _slurm_rpc_dump_partitions, size=1253 usec=10 ``` And from slurmd: ``` [2025-04-27T19:45:01.353] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.475] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.491] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.501] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.504] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.507] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.513] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.606] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.607] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:04.988] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.992] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.995] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.999] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.011] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.016] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.033] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.045] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.048] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.057] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.073] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.077] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.110] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.144] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.170] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.172] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.174] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.203] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.204] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.207] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.316] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.318] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.321] [59253.batch] debug: Handling REQUEST_GETPW ``` This level of debugging makes the logs pretty huge, but if seeing a whole log file is helpful, I can make something available. Any ideas on next steps for figuring out what’s going on? It seems like something is asking for authentication a whole lot, but it’s not clear to me what or why. We do use munge for Slurm authentication, and SSSD to work with LDAP for user authentication. -Jeremy Guillette — Jeremy Guillette Software Engineer, FAS Academic Technology | Academic Technology Harvard University Information Technology P: (617) 998-1826 | W: huit.harvard.edu (he/him/his)

4 6

Large memory jobs stuck Pending. Should use --time parameter?
by Mike 07 May '25

07 May '25

Greetings, We are new to Slurm and we are trying to better understand why we’re seeing high-mem jobs stuck in Pending state indefinitely. Smaller (mem) jobs in the queue will continue to pass by the high mem jobs even when we bump priority on a pending high-mem job way up. We have been reading over the backfill scheduling page and what we think we're seeing is that we need to require that users specify a --time parameter on their jobs so that Backfill works properly. None of our users specify a --time param because we have never required it. Is that what we need to require in order to fix this situation? From the backfill page: "Backfill scheduling is difficult without reasonable time limit estimates for jobs, but some configuration parameters that can help" and it goes on to list some config params that we have not set (DefaultTime, MaxTime, OverTimeLimit). We also see language such as, “Since the expected start time of pending jobs depends upon the expected completion time of running jobs, reasonably accurate time limits are important for backfill scheduling to work well.” So we suspect that we can achieve proper backfill scheduling by requiring that all users supply a "--time" parameter via a job submit plugin. Would that be a fair statement? Thank you in advance! -Mike Schor

4 4

Trying to using PMIx via srun for openmpi + pmix 5.0.7, PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 (but works with --mpi=pmix_v3?!)
by Dr. Thomas Orgis 06 May '25

06 May '25

Dear Slurm community, I am confused by the behaviour of a freshly built openmpi-5.0.7 with slurm-24.11.4. I can run a simple hello-world program via mpirun, but with really slow startup (a single process needing 1.6 s, 384 processes on two 192-core nodes need around half a minute). I guess there is a deeper issue here to work out with openmpi itself and why it needs 1.3 seconds to start a single process, even, outside a Slurm environment. So I investigate if running via srun changes things. Open-MPI docs recommend using mpirun and that has been traditionally our safe bet. But srun direct launch is supposed to work, too. I did duly build slurm-24.11.4 with ./configure --prefix=/syssw/slurm/24.11.4 \ --sysconfdir=/syssw/etc/slurm \ --with-munge=/syssw/munge/0.5.16 \ --with-hwloc=/syssw/hwloc/2.11.2 \ --disable-static --with-json \ --with-pmix=/syssw/pmix/3.2.5:/syssw/pmix/5.0.7 \ LDFLAGS=-Wl,--disable-new-dtags providing the two versions of pmix on our system currently. Now I am perplexed to observe that $ srun -vv --mpi=pmix_v5 -N 1 -n 1 mpihello does _not_ work, but produces c$ srun -vv -n 1 -N 1 --mpi=pmix_v5 mpihello srun: defined options srun: -------------------- -------------------- srun: (null) : n[164-165] srun: jobid : 671133 srun: job-name : interactive srun: mpi : pmix_v5 srun: nodes : 1 srun: ntasks : 1 srun: verbose : 2 srun: -------------------- -------------------- srun: end of defined options srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0022 srun: jobid 671133: nodes(2):`n[164-165]', cpu counts: 192(x2) srun: debug: requesting job 671133, user 99, nodes 1 including ((null)) srun: debug: cpus 1, tasks 1, name mpihello, relative 65534 srun: CpuBindType=none srun: debug: Entering slurm_step_launch srun: debug: mpi/pmix_v5: pmixp_abort_agent_start: (null) [0]: pmixp_agent.c:417: Abort agent port: 36505 srun: debug: mpi/pmix_v5: _pmix_abort_thread: (null) [0]: pmixp_agent.c:356: Start abort thread srun: debug: mpi/pmix_v5: mpi_p_client_prelaunch: (null) [0]: mpi_pmix.c:282: setup process mapping in srun srun: debug: Entering _msg_thr_create() srun: debug: initialized stdio listening socket, port 38681 srun: debug: Started IO server thread srun: debug: Entering _launch_tasks srun: launching StepId=671133.3 on host n165, 1 tasks: 0 srun: topology/tree: init: topology tree plugin loaded srun: debug: launch returned msg_rc=0 err=0 type=8001 srun: Node n165, 1 tasks started [n165:2287661] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and MPI will try to terminate your MPI job as well) [n165:2287661] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! srun: Received task exit notification for 1 task of StepId=671133.3 (status=0x0e00). srun: error: n165: task 0: Exited with exit code 14 srun: debug: task 0 done srun: debug: IO thread exiting srun: debug: mpi/pmix_v5: _conn_readable: (null) [0]: pmixp_agent.c:109: false, shutdown srun: debug: mpi/pmix_v5: _pmix_abort_thread: (null) [0]: pmixp_agent.c:363: Abort thread exit Observe the line [n165:2287661] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 anyone got an idea what that means, by what it's caused? And, what really confuses me, is that the test program _does_ work if I switch from pmix_v5 to pmix_v3: $ srun -vv -n 1 -N 1 --mpi=pmix_v3 mpihello srun: defined options srun: -------------------- -------------------- srun: (null) : n[164-165] srun: jobid : 671133 srun: job-name : interactive srun: mpi : pmix_v3 srun: nodes : 1 srun: ntasks : 1 srun: verbose : 2 srun: -------------------- -------------------- srun: end of defined options srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0022 srun: jobid 671133: nodes(2):`n[164-165]', cpu counts: 192(x2) srun: debug: requesting job 671133, user 99, nodes 1 including ((null)) srun: debug: cpus 1, tasks 1, name mpihello, relative 65534 srun: CpuBindType=none srun: debug: Entering slurm_step_launch srun: debug: mpi/pmix_v3: pmixp_abort_agent_start: (null) [0]: pmixp_agent.c:417: Abort agent port: 43737 srun: debug: mpi/pmix_v3: _pmix_abort_thread: (null) [0]: pmixp_agent.c:356: Start abort thread srun: debug: mpi/pmix_v3: mpi_p_client_prelaunch: (null) [0]: mpi_pmix.c:282: setup process mapping in srun srun: debug: Entering _msg_thr_create() srun: debug: initialized stdio listening socket, port 43805 srun: debug: Started IO server thread srun: debug: Entering _launch_tasks srun: launching StepId=671133.4 on host n164, 1 tasks: 0 srun: topology/tree: init: topology tree plugin loaded srun: debug: launch returned msg_rc=0 err=0 type=8001 srun: Node n164, 1 tasks started hello world from processor n164, rank 0 out of 1 srun: Received task exit notification for 1 task of StepId=671133.4 (status=0x0000). srun: n164: task 0: Completed srun: debug: task 0 done srun: debug: IO thread exiting srun: debug: mpi/pmix_v3: _conn_readable: (null) [0]: pmixp_agent.c:109: false, shutdown srun: debug: mpi/pmix_v3: _pmix_abort_thread: (null) [0]: pmixp_agent.c:363: Abort thread exit How can a PMIx 5 MPI even work with the pmix_v3 plugin? Why does it _not_ work with the pmix_v5 plugin? I am also curious why the plugins don't link to the respective libpmix (are they using dlopen for their dependencies? Why?). $ ldd /syssw/slurm/24.11.4/lib/slurm/mpi_pmix*.so /syssw/slurm/24.11.4/lib/slurm/mpi_pmix.so: linux-vdso.so.1 (0x00007ffd19ffb000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x000014d199e56000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014d199c70000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014d199b90000) /lib64/ld-linux-x86-64.so.2 (0x000014d199edd000) /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v3.so: linux-vdso.so.1 (0x00007ffd265f2000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x00001553902c8000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00001553900e2000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000155390002000) /lib64/ld-linux-x86-64.so.2 (0x0000155390350000) /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v5.so: linux-vdso.so.1 (0x00007ffd862b7000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x0000145adc36d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000145adc187000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000145adc0a7000) /lib64/ld-linux-x86-64.so.2 (0x0000145adc3f4000) But they do have the proper RPATH set up: $ readelf -d /syssw/slurm/24.11.4/lib/slurm/mpi_pmix*.so | grep -e ^File -e PATH File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/5.0.7/lib] File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v3.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/3.2.5/lib] File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v5.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/5.0.7/lib] Which is important, since libpmix doesn't get sensible SONAME versioning (supposing they are supposed to be separate ABIs): $ find /syssw/pmix/* -name 'libpmix.so*' /syssw/pmix/3.2.5/lib/libpmix.so /syssw/pmix/3.2.5/lib/libpmix.so.2.2.35 /syssw/pmix/3.2.5/lib/libpmix.so.2 /syssw/pmix/5.0.7/lib/libpmix.so /syssw/pmix/5.0.7/lib/libpmix.so.2.13.7 /syssw/pmix/5.0.7/lib/libpmix.so.2 It's all libpmix.so.2. My mpihello program uses the 5.0.7 one, at least: $ ldd mpihello linux-vdso.so.1 (0x00007fff1f13f000) libmpi.so.40 => /sw/env/gcc-13.3.0/openmpi/5.0.7/lib/libmpi.so.40 (0x000014ed89d95000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014ed89baf000) libopen-pal.so.80 => /sw/env/gcc-13.3.0/openmpi/5.0.7/lib/libopen-pal.so.80 (0x000014ed89a25000) libfabric.so.1 => /syssw/fabric/1.21.0/lib/libfabric.so.1 (0x000014ed8989b000) libefa.so.1 => /lib/x86_64-linux-gnu/libefa.so.1 (0x000014ed8988d000) libibverbs.so.1 => /lib/x86_64-linux-gnu/libibverbs.so.1 (0x000014ed8986c000) libpsm2.so.2 => /syssw/psm2/12.0.1/lib/libpsm2.so.2 (0x000014ed89804000) libatomic.so.1 => /sw/compiler/gcc-13.3.0/lib64/libatomic.so.1 (0x000014ed897fb000) libnl-route-3.so.200 => /lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x000014ed8976a000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x000014ed89745000) libpmix.so.2 => /syssw/pmix/5.0.7/lib/libpmix.so.2 (0x000014ed8951e000) libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x000014ed894e8000) libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x000014ed894e3000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x000014ed89486000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014ed893a6000) /lib64/ld-linux-x86-64.so.2 (0x000014ed8a0d8000) libnuma.so.1 => /lib/x86_64-linux-gnu/libnuma.so.1 (0x000014ed89397000) Can someone shed light on how the differing PMIx plugins are supposed to work? Can someone share a setup where pmix_v5 does work with openmpi 5.x? Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg

1 0

knl_generic.conf seems to be ignored and Slurm adopts default settings which specify Intel motherboard and syscfg - need help to figure out why Slurm is not reading my Dell knl_generic.conf parameters
by Bryan Johnston 01 May '25

01 May '25

Hi folks I'm in the process of standing up some Stampede2 Dell C3620p KNL nodes and I seem to be hitting a blind spot. I previously "successfully" configured KNL's on an Intel board (S7200AP), with OpenHPC and Rocky8. I say "successfully" because it works but evidently my latest troubleshooting has revealed that I may have been lucky rather than an expert KNL integrator :-) I thought I knew what I was doing, but after repeating my Intel KNL recipe with the Dell system, I have unearthed my ignorance with this wonderful (but deprecated) technology (anecdotally, the KNLs offer excellent performance and power efficiency for our workloads, particularly when contrasted with our alternative available hardware). The first discovery was the "syscfg" for Intel boards is not the same as the "syscfg" for Dell boards. I've since sorted this out. The second discovery was made while troubleshooting an issue that I'm hitting. After realising that the slurmd client nodes don't seem to be reading the "knl_generic.conf" parameters that are specified in /etc/slurm on the smshost (OpenHPC parlance for head node ... And it's a Slurm config less set up), I think my original Intel solution was working out of luck more than ingenuity. For reference , the Slurm configuration for KNL now includes: ``` NodeFeaturesPlugins=knl_generic DebugFlags=NodeFeatures GresTypes=hbm ``` And I've created a separate "knl_generic.conf" that points to the Dell specific tools and features. For the Dell system, slurmd seems to ignore my knl_generic.conf file and is drawing defaults from somewhere else. Slurm still considers SystemType to be Intel, SyscfgPath to be the default location, and SyscfgTimeout to be 1000. For Dell systems, Slurm needs to have SystemType=Dell and Timeout to be 1000. I don't understand why the nodes are not reading the knl_generic file - any help or clues would be appreciated. Here's my theory on what is happening: The Intel KNL system was successful by luck ... It probably exhibited the same ignore-the-config-file but ran default NodeFeatures for some generic knl_generic settings which are stored somewhere as default parameters. I must have just lucked out when I was using my Intel KNL system because it was using the defaults (that are compatible with Intel). If this assumption is correct, the Dell system is not working because it isn't compatible with the Intel defaults. Any clues on how to successfully invoke the config file (or better debuggingtechniwues to figure out why it isn't) would be appreciated. I can share journalctl feedback if necessary. For now, I've tried changing ownership of the config files to root:slurm, copied knl_generic.conf to the compute nodes' /etc/slurm/ and also tried to specify the config file by running (on the compute nodes) "slurmd" with "-f" ... No joy; if slurmd runs successfully (when I don't screw up some random experimental settings) then it always seems to ignore knl_generic.conf and loads some default settings from somewhere. A few questions: 1. Are there default settings stored somewhere? I might be barking up the wrong tree, although I've looked for files that may clash with the config file I've created but can't find any. 2. Is there a better way to force the knl_generic file to be loaded? 3. Is the configless Slurm somehow not reading the knl_generic file to the clients? I understand that all configuration files are read from the host server. Many thanks for any help! Regards / Groete / Sala(ni) Kahle - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Bryan Johnston Senior HPC Technologist II Lead: HPC Ecosystems Project HPCwire 2024's Outstanding Leader in HPC CHPC | www.chpc.ac.za | NICIS | nicis.ac.za Centre for High Performance Computing If you receive an email from me out of office hours for you, please do not feel obliged to respond during off-hours! Book time to meet with me<https://outlook.office.com/bookwithme/user/87af4906a703488386578f34e4473c74…>

1 1

Slurm and Kerberos/GSSAPI
by Burian, John 01 May '25

01 May '25

Does anyone have any experience with using Kerberos/GSSAPI and Slurm? I’m specifically wondering if there is a known mechanism for providing proper Kerberos credentials to Slurm batch jobs, such that those processes would be able to access a filesystem that requires Kerberos credentials. Some quick searching returned nothing useful. Interactive jobs have a similar problem, but I’m hoping that SSH credential forwarding can be leveraged there. I’m nothing like an expert in Kerberos, so forgive any apparent ignorance. Thanks, John

2 3

Can't specify multiple partitions when submitting GPU jobs
by milad＠earthspecies.org 29 Apr '25

29 Apr '25

Hi, I have defined a partition for each GPU type we have in the cluster. This was mainly because I have different Node types for each GPU type and I want to set `DefCpuPerGPU` `DefMemPerGPU` for each of them. Unfortunately one can't set them per node but can do that per partition. Now sometimes people don't care about the GPU type and would like any of the partitions to pick up the job. The `--partition` in `sbatch` does allow specifying multiple partitions and this works fine when I'm not specifying `--gpu`. However when I add do something like `sbatch -p A,B --gpus 1 script.sh` then I get "srun: job 6279 queued and waiting for resources" even though partition B does have a GPU to offer. Strangely if the first partition specified (i.e. A) had a free GPU it would allocate the GPU and run the job. Is this a bug? Perhaps related to this: https://groups.google.com/g/slurm-users/c/UOUVfkajUBQ

3 4

Re: Error "_check_core_range_matches_sock" when
by Gestió Servidors 29 Apr '25

29 Apr '25

Hello, Yes, I know system is single-socket, but in a similar computer, this configuration works fine... After some tests, I have realized that with SLURM 23.11.0 I can assign CPUs in "gres.conf" (and, then, use half CPU for one GPU and other half for other GPU), but with SLURM 24.11.4 I receive " _check_core_range_matches_sock". I have read "NEWS" file inside sources files in 24.11.4 tgz and it says: […] * Changes in Slurm 23.11.7 ========================== […] -- Fix issue that only left one thread on each core available when "CPUs=" is configured to total thread count on multi-threaded hardware and no other topology info ("Sockets=", "CoresPerSocket", etc.) is configured. […] So, I suppose, in version 23.11.7 SLURM corrected that behaviour. Could someone confirm that? Thanks.

1 0

Error "_check_core_range_matches_sock" when starting "slurmctld"
by Gestió Servidors 28 Apr '25

28 Apr '25

Hello, I have compiled SLURM-24.11.3 and I have configured two GPUs in my system (slurmctld and slurmd are running in the same computer). Computes has a old processor Intel i7 with 4 cores and 4 hyperthreading. Node is configured with "NodeName=mysystem CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7940 Gres=gpu:geforce_gtx_titan_x:1,gpu:geforce_gtx_titan_black:1". "lscpu" command returns: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 26 Model name: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz BIOS Model name: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz File gres.conf is: NodeName=mysystem Autodetect=off Name=gpu Type=geforce_gtx_titan_x File=/dev/nvidia0 CPUs=0-1 NodeName=mysystem Autodetect=off Name=gpu Type=geforce_gtx_titan_black File=/dev/nvidia1 CPUs=2-3 However, when I start daemon "slurmctld", system returns this error: [2025-04-28T09:35:41.003] error: _check_core_range_matches_sock: gres/gpu GRES core specification 0-1 for node aopcvis5 doesn't match socket boundaries. (Socket 0 is cores 0-3) [2025-04-28T09:35:41.003] error: Setting node aopcvis5 state to INVAL with reason:gres/gpu GRES core specification 0-1 for node aopcvis5 doesn't match socket boundaries. (Socket 0 is cores 0-3) Where is my configuration error? Thanks.

2 1

How can we put limits on interactive jobs?
by Ole Holm Nielsen 25 Apr '25

25 Apr '25

We would like to put limits on interactive jobs (started by salloc) so that users don't leave unused interactive jobs behind on the cluster by mistake. I can't offhand find any configurations that limit interactive jobs, such as enforcing a timelimit. Perhaps this could be done in job_submit.lua, but I couldn't find any job_desc parameters in the source code which would indicate if a job is interactive or not. Question: How do people limit interactive jobs, or identify orphaned jobs and kill them? Thanks a lot, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

4 4

Setting QoS with slurm 24.05.7
by Patrick Begou 25 Apr '25

25 Apr '25

Hi all, I'm trying to setup a QoS on a small 5 nodes cluster running slurm 24.05.7. My goal is to limit the resources on a (time x number of cores) strategy to avoid one large job requesting all the resources for too long time. I've read from https://slurm.schedmd.com/qos.html and some discussion but my setup is still not working. I think I need to set these informations: MaxCPUsPerJob=172800 MaxWallDurationPerJob=48:00:00 Flags=DenyOnLimit,OverPartQOS for: 12h max for 240 cores => (12*240*60=172800mn) no job can exceed 2 days do not accept jobs out of these limits. What I've done: 1) create the QoS: sudo sacctmgr add qos workflowlimit \ MaxWallDurationPerJob=48:00:00 \ MaxCPUsPerJob=172800 \ Flags=DenyOnLimit,OverPartQOS 2) Check sacctmgr show qos Name=workflowlimit format=Name%16,MaxTRES,MaxWall Name MaxTRES MaxWall ---------------- ------------- ----------- workflowlimit cpu=172800 2-00:00:00 3) Set the QoS for the account "most" which is the default account for the users: sudo sacctmgr modify account name=most set qos=workflowlimit 4) Check $ sacctmgr show assoc format=account,cluster,user,qos Account Cluster User QOS ---------- ---------- ---------- -------------------- root osorno normal root osorno root normal legi osorno normal most osorno workflowlimit most osorno begou workflowlimit 5) Modifiy slurm.conf with: AccountingStorageEnforce=limits,qos and propagate on the 5 nodes and the front end (done via Ansible) 6) Check clush -b -w osorno-fe,osorno,osorno-0-[0-4] 'grep AccountingStorageEnforce /etc/slurm/slurm.conf' --------------- osorno,osorno-0-[0-4],osorno-fe (7) --------------- AccountingStorageEnforce=limits,qos 7) restart slurmd on all the compute nodes and slurmctld + slurmdbd on the management node. But I can still request 400 cores for 24 hours: [begou@osorno ~]$ srun -n 400 -t 24:0:0 --pty bash bash-5.1$ squeue JOBID PARTITION NAME USER ST TIME START_TIME TIME_LIMIT CPUS NODELIST(REASON) 147 genoa bash begou R 0:03 2025-04-18T16:52:11 1-00:00:00 400 osorno-0-[0-4] So I must have missed something ? My partition (I've only one) in slurm.conf is: PartitionName=genoa State=UP Default=YES MaxTime=48:00:00 DefaultTime=24:00:00 Shared=YES OverSubscribe=NO Nodes=osorno-0-[0-4] Thanks Patrick

2 4

Job running slower when using Slurm
by Jeffrey Layton 24 Apr '25

24 Apr '25

Good morning, I'm running an NPB test, bt.C that is OpenMP and built using NV HPC SDK (version 25.1). I run it on a compute node by ssh-ing to the node. It runs in about 19.6 seconds. Then I run the code using a simple job: Command to submit job: sbatch --nodes=1 run-npb-omp The script run-npb-omp is the following: #!/bin/bash cd /home/.../NPB3.4-OMP/bin ./bt.C.x When I use Slurm, the job takes 482 seconds. Nothing really appears in the logs. It doesn't do any IO. No data is copied anywhere. I'm king of at a loss to figure out why. Any suggestions of where to look? Thanks! Jeff

4 9

Strange output of sshare
by frsc＠adm.ku.dk 23 Apr '25

23 Apr '25

Hi all, I'm trying to clean up and reconfigure fair share on a slurm 20.11.9 production cluster after some trial and error before I started looking into it. I don't know the full story and need to pick up here. Fair share is enabled with default settings and not customised. It looks a bit like it was enabled by accident and relevant options are undefined/at default. As a first step I would like to understand the current state. Executing sshare includes output like this (output shortened but complete for the two relevant accounts): Account User Partition RawShares NormShares RawUsage EffectvUsage FairShare -------- ---------- ------------ ---------- ----------- ----------- ------------- ---------- root 0.000000 1002501 1.000000 .root root 1 0.083333 0 0.000000 1.000000 .A1 1 0.083333 0 0.000000 ..A1 U1 P1 1 0.166667 0 0.000000 0.873684 ..A1 U2 P1 1 0.166667 0 0.000000 0.873684 ..A1 U3 P1 1 0.166667 0 0.000000 0.873684 ..A1 U3 P2 1 0.166667 0 0.000000 0.873684 ..A1 U4 P3 1 0.166667 0 0.000000 0.873684 ..A1 U4 P1 1 0.166667 0 1.000000 0.821053 <== What is going on here?? .A2 1 0.083333 0 0.000000 ..A2 U5 P2 1 0.142857 0 0.000000 1.000000 ..A2 U5 P1 1 0.142857 0 0.000000 1.000000 ..A2 U6 P4 1 0.142857 0 0.000000 1.000000 ..A2 U6 P1 1 0.142857 0 0.000000 1.000000 ..A2 U6 P2 1 0.142857 0 0.000000 1.000000 ..A2 U7 P1 1 0.142857 0 0.000000 1.000000 ..A2 U7 P2 1 0.142857 0 0.000000 1.000000 User U4 is not a member of any other account. I understand everything about this output except the line I marked. Both accounts A1 and A2 have zero usage, yet, for user U4 on partition P1 we have effective usage 1.0, screwing the fair share factors up for everyone in this account. As far as I can tell, both accounts should look identical with a fair share factor of 1 for every user. I'm grateful for any pointer for what to look for. Best regards, Frank

2 1

Slurm webhooks
by Davide DelVento 22 Apr '25

22 Apr '25

Happy Monday everybody, I've gotten a request to have Slurm notify users for the typical email things (job started, completed, failed, etc) with a REST API instead of email. This would allow notifications in MS Teams, Slack, or log stuff in some internal websites and things like that. As far as I can tell, Slurm does not support that, for example there was somebody who was looking for that on Galaxy and did not find a solution: https://help.galaxyproject.org/t/web-hook-post-to-external-url-when-job-beg… Is that indeed the case, as searching the web indicates? If Slurm does not support this, is there a workaround? For example, I'm thinking of installing a local SMTP server, or an alternative/dummy mailx program which instead of relaying emails as requested would post an encrypted web url thing using the information from the email. I am sure I could actually write such a software myself, but I don't have enough time to dedicate to the design, maintenance and debugging of such, so I am looking for something decent already in existence. A cursory web search did not find anything suitable, but perhaps I did not look in the appropriate places, because my gut feeling is that somebody must have already had such an itch to scratch! Any other ideas about alternative ways to accomplish this? Thanks

3 3

problem with slurmd dynamic mode with slurmctld in docker
by stef53864 22 Apr '25

22 Apr '25

hi everybody, i try to user dynamic mode with configless mode with slurm 24.11.3 and upgrade to 24.11.4 with slurmd and i found a problem. slurmctld is a container with docker, and my node is outside the container network . slurmctld register my ip with a function getpeeraddr on the slurmctld socket. but my ip connected to the socket come from the docker nat/bridge so slurmctld register my ip bridged ( not my real ip ) that is to say the docker gateway (172.20.0.1) *scontrol show node* *-------------------------* *NodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=172.20.0.1 NodeHostName=ltlsbubble1 Version=24.11.4* so the node go down after the "not pinging it" timeout i try to update the config *scontrol uupdate NodeName=ltlsbubble1* *NodeAddr=xx.xx.xx.xx* but a the first *scontrol reconfigure * it comes back to : *NodeAddr=172.20.0.1* in normal mode ------------------- *scontrol show nodeNodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=ltlsbubble1 NodeHostName=ltlsbubble1 Version=24.11.4* in normal mode NodeAddr is the same than NodeName , so it use DNS resolution for communication. to verify my hypothesis, i go to the c code of slurm, identify the register function and replace it with the same mechanism than normal node in src/slurmctld/node_mgr.c i replace : set_node_comm_name(node_ptr, *comm_name*, reg_msg->hostname); by set_node_comm_name(node_ptr, NULL, reg_msg->hostname); i rebuild slutmctld with this patch and try it with dynamic mode , it works like expected *scontrol show nodeNodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=ltlsbubble1 NodeHostName=ltlsbubble1 Version=24.1* no ip in nodeAddr , but only the nodename, so it use DNS resolution . the node works fine and no goes down for timeout ping so my question : can we have an option to force DNS resolution instead ip discover in Dynamic mode ? ( i try the option cloud_dns, but it not seems the purpose of this option) best regard, Stephane

1 0

pam_slurm_adopt ambiguity in instructions and with ssd
by Robert Kudyba 21 Apr '25

21 Apr '25

In the instructions for pam_slurm_adopt <https://slurm.schedmd.com/pam_slurm_adopt.html#ssh_config>, there are instructions such as: > > Add the following line to the appropriate file in /etc/pam.d, such as > system-auth or sshd (you may use either the "required" or "sufficient" > PAM control flag): This module is configurable. Add these options to the end of the > pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd > or system-auth): Assuming an OS like CentOS does this mean it should be put in both? slurm.conf on the node has: UsePAM yes slurm.conf has PrologFlags=contain and ProctrackType=proctrack/cgroup I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack. #%PAM-1.0 auth required pam_sepermit.so auth substack system-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so account include system-auth -account required pam_slurm_adopt.so so pam_sss.so is at the bottom of /etc/pam.d/sshd session optional pam_keyinit.so revoke session required pam_limits.so #-session optional pam_systemd.so session optional pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets: Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user(a)ouruni.EDU (ssh_gssapi_krb5_cmdok) Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth] Am I missing something?

3 5

Issue with Enforcing GPU Usage Limits in Slurm
by lyz＠simplehpc.com 16 Apr '25

16 Apr '25

Hi, I am currently encountering an issue with Slurm's GPU resource limitation. I have attempted to restrict the number of GPUs a user can utilize by executing the following command: sacctmgr modify user lyz set MaxTRES=gres/gpu=2 This command is intended to limit user 'lyz' to using a maximum of 2 GPUs. However, when the user submits a job using srun, specifying CUDA 0, 1, 2, and 3 in the job script, or os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3", the job still utilizes all 4 GPUs during execution. This indicates that the GPU usage limit is not being enforced as expected. How can I resolve this situation.

4 14

pam_slurm_adopt and multiple jobs on the same worker node
by Massimo Sgaravatto 14 Apr '25

14 Apr '25

Dear all With the pam_slurm_adopt module as far as I understand you can ssh to a worker node if there is at least a job running on the node by that user. If there are multiple jobs, if I am not wrong you will be "mapped" to the last job started on the node. And, if you are using cgroups, you will be "confined" to the resources assigned to this "last" job. Is it possible in some way to specify the job to be mapped, in case there are multiple jobs for that user on the same node ? Thanks, Massimo

2 2

sllurmrestd via unix socket
by Brian Andrus 10 Apr '25

10 Apr '25

All, Maybe someone has seen this. I have slurmrestd running listening on port 8081 as well as a unix socket at /run/slurmrestd/slurmrestd.sock I am able to query the port with curl and do a ping. Everything seems fine. Other commands work as well. "pings": [ { "hostname": "head01", "pinged": "UP", "responding": true, "latency": 1605, "mode": "primary", "primary": true }, { "hostname": "head02", "pinged": "UP", "responding": true, "latency": 2353, "mode": "backup", "primary": false } When I try doing so via socket, however, my two head nodes show 'DOWN' : "pings": [ { "hostname": "head01", "pinged": "DOWN", "latency": 11784, "mode": "primary" }, { "hostname": "head02", "pinged": "DOWN", "latency": 12668, "mode": "backup" } Other commands fail with: "error_number": 1007, "error": "Protocol authentication error", I'll admit, I don't usually use sockets, so I could easily be overlooking something there. Permissions on the socket look right. I am getting json back, so it is connecting. Note: slurmrestd is running under it's own user (not root and not slurmuser). Any ideas? Thanks in advance, Brian Andrus

1 0

Minimum cpu cores per node partition level configuration
by Jeherul Islam 10 Apr '25

10 Apr '25

Dear All, I need to configure the slurm so the user must take a certain minimum number of CPU cores for a particular partition(not system-wide). Otherwise, the job must not run. Any suggestions will be highly appreciated. With Thanks and Regards -- Jeherul Islam

4 4

weird sacct behavior?
by Pierre Abele 10 Apr '25

10 Apr '25

Hi everyone, I am currently stuck with an sacct issue and would appreciate any help/hints/ideas: My users cannot retrieve job data from their currently running jobs through sacct anymore. Running sacct -a as root also reproduces this issue: It does not show running jobs, but both sacct -j <JobID> and squeue -j <JobID> do. AFAICT, this is not intended behavior (?). Also including longer time windows witch sacct -S ... -E did not help. root@slurmmaster:~# sacct -a | grep 154415 # this returns nothing root@slurmmaster:~# sacct -j 154415 JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- 154415 allocation primevo 0 PENDING 0:0 154415.batch batch primevo 2 RUNNING 0:0 154415.exte+ extern primevo 2 RUNNING 0:0 root@slurmmaster:~# squeue -j 154415 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 154415 standard genedrop username R 1:31 1 hpc020 Also, possibly related, we had a slurmdbd crash before this changed. We run Ubuntu Server 24.04 LTS with Slurm 24.05.4, using a MariaDB accounting database hosted on the same machine as the Slurm controller. Does anyone here have any ideas? Best, Pierre -- Pierre Abele, M.Sc. HPC Administrator Max-Planck-Institute for Evolutionary Anthropology Department of Primate Behavior and Evolution Deutscher Platz 6 04103 Leipzig Room: U2.80 E-Mail: pierre_abele(a)eva.mpg.de Phone: +49 (0) 341 3550 245

1 0

2025

2024

slurm-users ----- 2025 ----- June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users