- slurm-users - lists.schedmd.com

Trying to using PMIx via srun for openmpi + pmix 5.0.7, PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 (but works with --mpi=pmix_v3?!)
by Dr. Thomas Orgis 06 May '25

06 May '25

Dear Slurm community, I am confused by the behaviour of a freshly built openmpi-5.0.7 with slurm-24.11.4. I can run a simple hello-world program via mpirun, but with really slow startup (a single process needing 1.6 s, 384 processes on two 192-core nodes need around half a minute). I guess there is a deeper issue here to work out with openmpi itself and why it needs 1.3 seconds to start a single process, even, outside a Slurm environment. So I investigate if running via srun changes things. … [View More]Open-MPI docs recommend using mpirun and that has been traditionally our safe bet. But srun direct launch is supposed to work, too. I did duly build slurm-24.11.4 with ./configure --prefix=/syssw/slurm/24.11.4 \ --sysconfdir=/syssw/etc/slurm \ --with-munge=/syssw/munge/0.5.16 \ --with-hwloc=/syssw/hwloc/2.11.2 \ --disable-static --with-json \ --with-pmix=/syssw/pmix/3.2.5:/syssw/pmix/5.0.7 \ LDFLAGS=-Wl,--disable-new-dtags providing the two versions of pmix on our system currently. Now I am perplexed to observe that $ srun -vv --mpi=pmix_v5 -N 1 -n 1 mpihello does _not_ work, but produces c$ srun -vv -n 1 -N 1 --mpi=pmix_v5 mpihello srun: defined options srun: -------------------- -------------------- srun: (null) : n[164-165] srun: jobid : 671133 srun: job-name : interactive srun: mpi : pmix_v5 srun: nodes : 1 srun: ntasks : 1 srun: verbose : 2 srun: -------------------- -------------------- srun: end of defined options srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0022 srun: jobid 671133: nodes(2):`n[164-165]', cpu counts: 192(x2) srun: debug: requesting job 671133, user 99, nodes 1 including ((null)) srun: debug: cpus 1, tasks 1, name mpihello, relative 65534 srun: CpuBindType=none srun: debug: Entering slurm_step_launch srun: debug: mpi/pmix_v5: pmixp_abort_agent_start: (null) [0]: pmixp_agent.c:417: Abort agent port: 36505 srun: debug: mpi/pmix_v5: _pmix_abort_thread: (null) [0]: pmixp_agent.c:356: Start abort thread srun: debug: mpi/pmix_v5: mpi_p_client_prelaunch: (null) [0]: mpi_pmix.c:282: setup process mapping in srun srun: debug: Entering _msg_thr_create() srun: debug: initialized stdio listening socket, port 38681 srun: debug: Started IO server thread srun: debug: Entering _launch_tasks srun: launching StepId=671133.3 on host n165, 1 tasks: 0 srun: topology/tree: init: topology tree plugin loaded srun: debug: launch returned msg_rc=0 err=0 type=8001 srun: Node n165, 1 tasks started [n165:2287661] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and MPI will try to terminate your MPI job as well) [n165:2287661] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! srun: Received task exit notification for 1 task of StepId=671133.3 (status=0x0e00). srun: error: n165: task 0: Exited with exit code 14 srun: debug: task 0 done srun: debug: IO thread exiting srun: debug: mpi/pmix_v5: _conn_readable: (null) [0]: pmixp_agent.c:109: false, shutdown srun: debug: mpi/pmix_v5: _pmix_abort_thread: (null) [0]: pmixp_agent.c:363: Abort thread exit Observe the line [n165:2287661] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 anyone got an idea what that means, by what it's caused? And, what really confuses me, is that the test program _does_ work if I switch from pmix_v5 to pmix_v3: $ srun -vv -n 1 -N 1 --mpi=pmix_v3 mpihello srun: defined options srun: -------------------- -------------------- srun: (null) : n[164-165] srun: jobid : 671133 srun: job-name : interactive srun: mpi : pmix_v3 srun: nodes : 1 srun: ntasks : 1 srun: verbose : 2 srun: -------------------- -------------------- srun: end of defined options srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0022 srun: jobid 671133: nodes(2):`n[164-165]', cpu counts: 192(x2) srun: debug: requesting job 671133, user 99, nodes 1 including ((null)) srun: debug: cpus 1, tasks 1, name mpihello, relative 65534 srun: CpuBindType=none srun: debug: Entering slurm_step_launch srun: debug: mpi/pmix_v3: pmixp_abort_agent_start: (null) [0]: pmixp_agent.c:417: Abort agent port: 43737 srun: debug: mpi/pmix_v3: _pmix_abort_thread: (null) [0]: pmixp_agent.c:356: Start abort thread srun: debug: mpi/pmix_v3: mpi_p_client_prelaunch: (null) [0]: mpi_pmix.c:282: setup process mapping in srun srun: debug: Entering _msg_thr_create() srun: debug: initialized stdio listening socket, port 43805 srun: debug: Started IO server thread srun: debug: Entering _launch_tasks srun: launching StepId=671133.4 on host n164, 1 tasks: 0 srun: topology/tree: init: topology tree plugin loaded srun: debug: launch returned msg_rc=0 err=0 type=8001 srun: Node n164, 1 tasks started hello world from processor n164, rank 0 out of 1 srun: Received task exit notification for 1 task of StepId=671133.4 (status=0x0000). srun: n164: task 0: Completed srun: debug: task 0 done srun: debug: IO thread exiting srun: debug: mpi/pmix_v3: _conn_readable: (null) [0]: pmixp_agent.c:109: false, shutdown srun: debug: mpi/pmix_v3: _pmix_abort_thread: (null) [0]: pmixp_agent.c:363: Abort thread exit How can a PMIx 5 MPI even work with the pmix_v3 plugin? Why does it _not_ work with the pmix_v5 plugin? I am also curious why the plugins don't link to the respective libpmix (are they using dlopen for their dependencies? Why?). $ ldd /syssw/slurm/24.11.4/lib/slurm/mpi_pmix*.so /syssw/slurm/24.11.4/lib/slurm/mpi_pmix.so: linux-vdso.so.1 (0x00007ffd19ffb000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x000014d199e56000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014d199c70000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014d199b90000) /lib64/ld-linux-x86-64.so.2 (0x000014d199edd000) /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v3.so: linux-vdso.so.1 (0x00007ffd265f2000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x00001553902c8000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00001553900e2000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000155390002000) /lib64/ld-linux-x86-64.so.2 (0x0000155390350000) /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v5.so: linux-vdso.so.1 (0x00007ffd862b7000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x0000145adc36d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000145adc187000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000145adc0a7000) /lib64/ld-linux-x86-64.so.2 (0x0000145adc3f4000) But they do have the proper RPATH set up: $ readelf -d /syssw/slurm/24.11.4/lib/slurm/mpi_pmix*.so | grep -e ^File -e PATH File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/5.0.7/lib] File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v3.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/3.2.5/lib] File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v5.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/5.0.7/lib] Which is important, since libpmix doesn't get sensible SONAME versioning (supposing they are supposed to be separate ABIs): $ find /syssw/pmix/* -name 'libpmix.so*' /syssw/pmix/3.2.5/lib/libpmix.so /syssw/pmix/3.2.5/lib/libpmix.so.2.2.35 /syssw/pmix/3.2.5/lib/libpmix.so.2 /syssw/pmix/5.0.7/lib/libpmix.so /syssw/pmix/5.0.7/lib/libpmix.so.2.13.7 /syssw/pmix/5.0.7/lib/libpmix.so.2 It's all libpmix.so.2. My mpihello program uses the 5.0.7 one, at least: $ ldd mpihello linux-vdso.so.1 (0x00007fff1f13f000) libmpi.so.40 => /sw/env/gcc-13.3.0/openmpi/5.0.7/lib/libmpi.so.40 (0x000014ed89d95000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014ed89baf000) libopen-pal.so.80 => /sw/env/gcc-13.3.0/openmpi/5.0.7/lib/libopen-pal.so.80 (0x000014ed89a25000) libfabric.so.1 => /syssw/fabric/1.21.0/lib/libfabric.so.1 (0x000014ed8989b000) libefa.so.1 => /lib/x86_64-linux-gnu/libefa.so.1 (0x000014ed8988d000) libibverbs.so.1 => /lib/x86_64-linux-gnu/libibverbs.so.1 (0x000014ed8986c000) libpsm2.so.2 => /syssw/psm2/12.0.1/lib/libpsm2.so.2 (0x000014ed89804000) libatomic.so.1 => /sw/compiler/gcc-13.3.0/lib64/libatomic.so.1 (0x000014ed897fb000) libnl-route-3.so.200 => /lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x000014ed8976a000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x000014ed89745000) libpmix.so.2 => /syssw/pmix/5.0.7/lib/libpmix.so.2 (0x000014ed8951e000) libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x000014ed894e8000) libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x000014ed894e3000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x000014ed89486000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014ed893a6000) /lib64/ld-linux-x86-64.so.2 (0x000014ed8a0d8000) libnuma.so.1 => /lib/x86_64-linux-gnu/libnuma.so.1 (0x000014ed89397000) Can someone shed light on how the differing PMIx plugins are supposed to work? Can someone share a setup where pmix_v5 does work with openmpi 5.x? Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg [View Less]

1 0

knl_generic.conf seems to be ignored and Slurm adopts default settings which specify Intel motherboard and syscfg - need help to figure out why Slurm is not reading my Dell knl_generic.conf parameters
by Bryan Johnston 01 May '25

01 May '25

Hi folks I'm in the process of standing up some Stampede2 Dell C3620p KNL nodes and I seem to be hitting a blind spot. I previously "successfully" configured KNL's on an Intel board (S7200AP), with OpenHPC and Rocky8. I say "successfully" because it works but evidently my latest troubleshooting has revealed that I may have been lucky rather than an expert KNL integrator :-) I thought I knew what I was doing, but after repeating my Intel KNL recipe with the Dell system, I have unearthed my … [View More]ignorance with this wonderful (but deprecated) technology (anecdotally, the KNLs offer excellent performance and power efficiency for our workloads, particularly when contrasted with our alternative available hardware). The first discovery was the "syscfg" for Intel boards is not the same as the "syscfg" for Dell boards. I've since sorted this out. The second discovery was made while troubleshooting an issue that I'm hitting. After realising that the slurmd client nodes don't seem to be reading the "knl_generic.conf" parameters that are specified in /etc/slurm on the smshost (OpenHPC parlance for head node ... And it's a Slurm config less set up), I think my original Intel solution was working out of luck more than ingenuity. For reference , the Slurm configuration for KNL now includes: ``` NodeFeaturesPlugins=knl_generic DebugFlags=NodeFeatures GresTypes=hbm ``` And I've created a separate "knl_generic.conf" that points to the Dell specific tools and features. For the Dell system, slurmd seems to ignore my knl_generic.conf file and is drawing defaults from somewhere else. Slurm still considers SystemType to be Intel, SyscfgPath to be the default location, and SyscfgTimeout to be 1000. For Dell systems, Slurm needs to have SystemType=Dell and Timeout to be 1000. I don't understand why the nodes are not reading the knl_generic file - any help or clues would be appreciated. Here's my theory on what is happening: The Intel KNL system was successful by luck ... It probably exhibited the same ignore-the-config-file but ran default NodeFeatures for some generic knl_generic settings which are stored somewhere as default parameters. I must have just lucked out when I was using my Intel KNL system because it was using the defaults (that are compatible with Intel). If this assumption is correct, the Dell system is not working because it isn't compatible with the Intel defaults. Any clues on how to successfully invoke the config file (or better debuggingtechniwues to figure out why it isn't) would be appreciated. I can share journalctl feedback if necessary. For now, I've tried changing ownership of the config files to root:slurm, copied knl_generic.conf to the compute nodes' /etc/slurm/ and also tried to specify the config file by running (on the compute nodes) "slurmd" with "-f" ... No joy; if slurmd runs successfully (when I don't screw up some random experimental settings) then it always seems to ignore knl_generic.conf and loads some default settings from somewhere. A few questions: 1. Are there default settings stored somewhere? I might be barking up the wrong tree, although I've looked for files that may clash with the config file I've created but can't find any. 2. Is there a better way to force the knl_generic file to be loaded? 3. Is the configless Slurm somehow not reading the knl_generic file to the clients? I understand that all configuration files are read from the host server. Many thanks for any help! Regards / Groete / Sala(ni) Kahle - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Bryan Johnston Senior HPC Technologist II Lead: HPC Ecosystems Project HPCwire 2024's Outstanding Leader in HPC CHPC | www.chpc.ac.za | NICIS | nicis.ac.za Centre for High Performance Computing If you receive an email from me out of office hours for you, please do not feel obliged to respond during off-hours! Book time to meet with me<https://outlook.office.com/bookwithme/user/87af4906a703488386578f34e4473c74…> [View Less]

1 1

Slurm and Kerberos/GSSAPI
by Burian, John 01 May '25

01 May '25

Does anyone have any experience with using Kerberos/GSSAPI and Slurm? I’m specifically wondering if there is a known mechanism for providing proper Kerberos credentials to Slurm batch jobs, such that those processes would be able to access a filesystem that requires Kerberos credentials. Some quick searching returned nothing useful. Interactive jobs have a similar problem, but I’m hoping that SSH credential forwarding can be leveraged there. I’m nothing like an expert in Kerberos, so forgive … [View More]

2 3

Can't specify multiple partitions when submitting GPU jobs
by milad＠earthspecies.org 29 Apr '25

29 Apr '25

Hi, I have defined a partition for each GPU type we have in the cluster. This was mainly because I have different Node types for each GPU type and I want to set `DefCpuPerGPU` `DefMemPerGPU` for each of them. Unfortunately one can't set them per node but can do that per partition. Now sometimes people don't care about the GPU type and would like any of the partitions to pick up the job. The `--partition` in `sbatch` does allow specifying multiple partitions and this works fine when I'm not … [View More]

3 4

Re: Error "_check_core_range_matches_sock" when
by Gestió Servidors 29 Apr '25

29 Apr '25

Hello, Yes, I know system is single-socket, but in a similar computer, this configuration works fine... After some tests, I have realized that with SLURM 23.11.0 I can assign CPUs in "gres.conf" (and, then, use half CPU for one GPU and other half for other GPU), but with SLURM 24.11.4 I receive " _check_core_range_matches_sock". I have read "NEWS" file inside sources files in 24.11.4 tgz and it says: […] * Changes in Slurm 23.11.7 ========================== […] -- Fix issue that only … [View More]

1 0

Error "_check_core_range_matches_sock" when starting "slurmctld"
by Gestió Servidors 28 Apr '25

28 Apr '25

Hello, I have compiled SLURM-24.11.3 and I have configured two GPUs in my system (slurmctld and slurmd are running in the same computer). Computes has a old processor Intel i7 with 4 cores and 4 hyperthreading. Node is configured with "NodeName=mysystem CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7940 Gres=gpu:geforce_gtx_titan_x:1,gpu:geforce_gtx_titan_black:1". "lscpu" command returns: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte … [View More]

2 1

How can we put limits on interactive jobs?
by Ole Holm Nielsen 25 Apr '25

25 Apr '25

We would like to put limits on interactive jobs (started by salloc) so that users don't leave unused interactive jobs behind on the cluster by mistake. I can't offhand find any configurations that limit interactive jobs, such as enforcing a timelimit. Perhaps this could be done in job_submit.lua, but I couldn't find any job_desc parameters in the source code which would indicate if a job is interactive or not. Question: How do people limit interactive jobs, or identify orphaned jobs … [View More]

4 4

Setting QoS with slurm 24.05.7
by Patrick Begou 25 Apr '25

25 Apr '25

Hi all, I'm trying to setup a QoS on a small 5 nodes cluster running slurm 24.05.7. My goal is to limit the resources on a (time x number of cores) strategy to avoid one large job requesting all the resources for too long time. I've read from https://slurm.schedmd.com/qos.html and some discussion but my setup is still not working. I think I need to set these informations: MaxCPUsPerJob=172800 MaxWallDurationPerJob=48:00:00 Flags=DenyOnLimit,OverPartQOS for: 12h max for 240 cores => (… [View More]12*240*60=172800mn) no job can exceed 2 days do not accept jobs out of these limits. What I've done: 1) create the QoS: sudo sacctmgr add qos workflowlimit \ MaxWallDurationPerJob=48:00:00 \ MaxCPUsPerJob=172800 \ Flags=DenyOnLimit,OverPartQOS 2) Check sacctmgr show qos Name=workflowlimit format=Name%16,MaxTRES,MaxWall Name MaxTRES MaxWall ---------------- ------------- ----------- workflowlimit cpu=172800 2-00:00:00 3) Set the QoS for the account "most" which is the default account for the users: sudo sacctmgr modify account name=most set qos=workflowlimit 4) Check $ sacctmgr show assoc format=account,cluster,user,qos Account Cluster User QOS ---------- ---------- ---------- -------------------- root osorno normal root osorno root normal legi osorno normal most osorno workflowlimit most osorno begou workflowlimit 5) Modifiy slurm.conf with: AccountingStorageEnforce=limits,qos and propagate on the 5 nodes and the front end (done via Ansible) 6) Check clush -b -w osorno-fe,osorno,osorno-0-[0-4] 'grep AccountingStorageEnforce /etc/slurm/slurm.conf' --------------- osorno,osorno-0-[0-4],osorno-fe (7) --------------- AccountingStorageEnforce=limits,qos 7) restart slurmd on all the compute nodes and slurmctld + slurmdbd on the management node. But I can still request 400 cores for 24 hours: [begou@osorno ~]$ srun -n 400 -t 24:0:0 --pty bash bash-5.1$ squeue JOBID PARTITION NAME USER ST TIME START_TIME TIME_LIMIT CPUS NODELIST(REASON) 147 genoa bash begou R 0:03 2025-04-18T16:52:11 1-00:00:00 400 osorno-0-[0-4] So I must have missed something ? My partition (I've only one) in slurm.conf is: PartitionName=genoa State=UP Default=YES MaxTime=48:00:00 DefaultTime=24:00:00 Shared=YES OverSubscribe=NO Nodes=osorno-0-[0-4] Thanks Patrick [View Less]

2 4

Job running slower when using Slurm
by Jeffrey Layton 24 Apr '25

24 Apr '25

Good morning, I'm running an NPB test, bt.C that is OpenMP and built using NV HPC SDK (version 25.1). I run it on a compute node by ssh-ing to the node. It runs in about 19.6 seconds. Then I run the code using a simple job: Command to submit job: sbatch --nodes=1 run-npb-omp The script run-npb-omp is the following: #!/bin/bash cd /home/.../NPB3.4-OMP/bin ./bt.C.x When I use Slurm, the job takes 482 seconds. Nothing really appears in the logs. It doesn't do any IO. No data is copied … [View More]

4 9

Strange output of sshare
by frsc＠adm.ku.dk 23 Apr '25

23 Apr '25

Hi all, I'm trying to clean up and reconfigure fair share on a slurm 20.11.9 production cluster after some trial and error before I started looking into it. I don't know the full story and need to pick up here. Fair share is enabled with default settings and not customised. It looks a bit like it was enabled by accident and relevant options are undefined/at default. As a first step I would like to understand the current state. Executing sshare includes output like this (output shortened but … [View More]complete for the two relevant accounts): Account User Partition RawShares NormShares RawUsage EffectvUsage FairShare -------- ---------- ------------ ---------- ----------- ----------- ------------- ---------- root 0.000000 1002501 1.000000 .root root 1 0.083333 0 0.000000 1.000000 .A1 1 0.083333 0 0.000000 ..A1 U1 P1 1 0.166667 0 0.000000 0.873684 ..A1 U2 P1 1 0.166667 0 0.000000 0.873684 ..A1 U3 P1 1 0.166667 0 0.000000 0.873684 ..A1 U3 P2 1 0.166667 0 0.000000 0.873684 ..A1 U4 P3 1 0.166667 0 0.000000 0.873684 ..A1 U4 P1 1 0.166667 0 1.000000 0.821053 <== What is going on here?? .A2 1 0.083333 0 0.000000 ..A2 U5 P2 1 0.142857 0 0.000000 1.000000 ..A2 U5 P1 1 0.142857 0 0.000000 1.000000 ..A2 U6 P4 1 0.142857 0 0.000000 1.000000 ..A2 U6 P1 1 0.142857 0 0.000000 1.000000 ..A2 U6 P2 1 0.142857 0 0.000000 1.000000 ..A2 U7 P1 1 0.142857 0 0.000000 1.000000 ..A2 U7 P2 1 0.142857 0 0.000000 1.000000 User U4 is not a member of any other account. I understand everything about this output except the line I marked. Both accounts A1 and A2 have zero usage, yet, for user U4 on partition P1 we have effective usage 1.0, screwing the fair share factors up for everyone in this account. As far as I can tell, both accounts should look identical with a fair share factor of 1 for every user. I'm grateful for any pointer for what to look for. Best regards, Frank [View Less]

2 1

2025

2024

slurm-users ----- 2025 ----- June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users