- slurm-users - lists.schedmd.com

Invalid/incorrect gres.conf syntax
by Gestió Servidors 20 May '24

20 May '24

Hello, I have configured my "gres.conf" in this way: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 node-gpu-1 and node-gpu-2 are two systems with two sockets; node-gpu-3 and node-gpu-4 have only one socket. In my "slurm.conf" I have these lines: AccountingStorageTRES=gres/gpu SelectType=select/cons_tres GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 However, slurmctld reports warning logs about "error syntax in Cores attribute" in gres.conf. Where is the syntax error? Thanks a lot!

1 0

Apply an specific QoS to all users that belongs to an specific account
by Gestió Servidors 20 May '24

20 May '24

Hi, I would like to know if it is possible to apply an specific QoS to all users that belongs to an specific account. For example, I have created some new users "user_XX" and, also, I have created their new accounts in SLURM with "sacctmgr create account name=Test" and "sacctmgr create user name=user_XX DefaultAccount=Test". After it, I have changed default QoS "normal" to a new QoS "minimal" (with some limits) to account "Test" (sacctmgr modify account where name=Test set qos=minimal), but what I have seen is that users "user_XX" that belongs to "Test" account continue in QoS "normal" (QoS by default), so it seems that users have not inherited QoS applied to their account. Is there any way to do this and am I doing something wrong? Thanks.

1 0

srun weirdness
by Dj Merrill 17 May '24

17 May '24

I'm running into a strange issue and I'm hoping another set of brains looking at this might help. I would appreciate any feedback. I have two Slurm Clusters. The first cluster is running Slurm 21.08.8 on Rocky Linux 8.9 machines. The second cluster is running Slurm 23.11.6 on Rocky Linux 9.4 machines. This works perfectly fine on the first cluster: $ srun --mem=32G --pty /bin/bash srun: job 93911 queued and waiting for resources srun: job 93911 has been allocated resources and on the resulting shell on the compute node: $ /mnt/local/ollama/ollama help and the ollama help message appears as expected. However, on the second cluster: $ srun --mem=32G --pty /bin/bash srun: job 3 queued and waiting for resources srun: job 3 has been allocated resources and on the resulting shell on the compute node: $ /mnt/local/ollama/ollama help fatal error: failed to reserve page summary memory runtime stack: runtime.throw({0x1240c66?, 0x154fa39a1008?}) runtime/panic.go:1023 +0x5c fp=0x7ffe6be32648 sp=0x7ffe6be32618 pc=0x4605dc runtime.(*pageAlloc).sysInit(0x127b47e8, 0xf8?) runtime/mpagealloc_64bit.go:81 +0x11c fp=0x7ffe6be326b8 sp=0x7ffe6be32648 pc=0x456b7c runtime.(*pageAlloc).init(0x127b47e8, 0x127b47e0, 0x128d88f8, 0x0) runtime/mpagealloc.go:320 +0x85 fp=0x7ffe6be326e8 sp=0x7ffe6be326b8 pc=0x454565 runtime.(*mheap).init(0x127b47e0) runtime/mheap.go:769 +0x165 fp=0x7ffe6be32720 sp=0x7ffe6be326e8 pc=0x451885 runtime.mallocinit() runtime/malloc.go:454 +0xd7 fp=0x7ffe6be32758 sp=0x7ffe6be32720 pc=0x434f97 runtime.schedinit() runtime/proc.go:785 +0xb7 fp=0x7ffe6be327d0 sp=0x7ffe6be32758 pc=0x464397 runtime.rt0_go() runtime/asm_amd64.s:349 +0x11c fp=0x7ffe6be327d8 sp=0x7ffe6be327d0 pc=0x49421c If I ssh directly to the same node on that second cluster (skipping Slurm entirely), and run the same "/mnt/local/ollama/ollama help" command, it works perfectly fine. My first thought was that it might be related to cgroups. I switched the second cluster from cgroups v2 to v1 and tried again, no difference. I tried disabling cgroups on the second cluster by removing all cgroups references in the slurm.conf file but that also made no difference. My guess is something changed with regards to srun between these two Slurm versions, but I'm not sure what. Any thoughts on what might be happening and/or a way to get this to work on the second cluster? Essentially I need a way to request an interactive shell through Slurm that is associated with the requested resources. Should we be using something other than srun for this? Thank you, -Dj

6 12

Slurm DB upgrade failure behavior
by Yuengling, Philip J. 17 May '24

17 May '24

Hi everyone, I'm writing up some Ansible code to manage Slurm software updates, and I haven't found any documentation about slurmdbd behavior if the mysql/mariadb database doesn't upgrade successfully. What I do know is that if it is sucessful I can expect to see "Conversion done: success!" in the slurmdbd log. This is good, but minor updates do not update the database as far as I know. If the Slurm database cannot upgrade upon an update, does it always shut down with a fatal error? What other behaviors should I look for if there is a failure? Cheers, Phil Y

2 1

Removing safely a node
by Ratnasamy, Fritz 17 May '24

17 May '24

Hi, What is the "official" process to remove nodes safely? I have drained the nodes so jobs are completed and put them in down state after they are completely drained. I edited the slurm.conf file to remove the nodes. After some time, I can see that the nodes were removed from the partition with the command sinfo However, I was told I might need to restart the service slurmctld, do you know if it is necessary? Should I also run scontrol reconfig? Best, *Fritz Ratnasamy* Data Scientist Information Technology

3 2

Best practice for jobs resuming from suspended state
by Paul Jones 16 May '24

16 May '24

Hi, We use PreemptMode and PriorityTier within Slurm to suspend low priority jobs when more urgent work needs to be done. This generally works well, but on occasion resumed jobs fail to restart - which is to say Slurm sets the job status to running but the actual code doesn't recover from being suspended. Technically everything is working as expected, but I wondered if there was any best practice to pass onto users about how to cope with this state? Obviously not a direct Slurm question, but wondered if others had experience with this and any advice on how best to limit the impact? Thanks, Paul --

2 1

Location of Slurm source packages?
by Jeffrey Layton 15 May '24

15 May '24

Good morning, I have an Ubuntu 22.04 server where I installed Slurm from the Ubuntu packages. I now want to install pyxis but it says I need the Slurm sources. In Ubuntu 22.04, is there a package that has the source code? How to download the sources I need from github? Thanks! Jeff

2 2

Re: Location of Slurm source packages?
by Lloyd Brown 15 May '24

15 May '24

Jeff, I'm not sure what version is in the Ubuntu packages, as I don't think they're provided by SchedMD, and I'm having trouble finding the right one on packages.ubuntu.com. Having said that, SchedMD is pretty good about using tags in their github repo (https://github.com/schedmd/slurm), to represent the releases. For example, the "slurm-23-11-6-1" tag corresponds to release 23.11.6. It's pretty straightforward to clone the repo, and do something like "git checkout -b MY_LOCAL_BRANCH_NAME TAG_NAME" to get the version you're after. Lloyd -- Lloyd Brown HPC Systems Administrator Office of Research Computing Brigham Young University http://rc.byu.edu On 5/15/24 08:35, Jeffrey Layton via slurm-users wrote: > Good morning, > > I have an Ubuntu 22.04 server where I installed Slurm from the Ubuntu > packages. I now want to install pyxis but it says I need the Slurm > sources. In Ubuntu 22.04, is there a package that has the source code? > How to download the sources I need from github? > > Thanks! > > Jeff

3 3

Re: Location of Slurm source packages?
by Lloyd Brown 15 May '24

15 May '24

Jeff, Dang. That's really old. I'm not sure I would run one that old, to be honest. Too many missing security fixes and added features. It's never been that hard to do a 'git clone' and the normal configure/make/make install process with slurm. Someone else made me aware of this, in case it's easier: https://slurm.schedmd.com/quickstart_admin.html#debuild Lloyd On 5/15/24 08:57, Jeffrey Layton wrote: > Lloyd, > > Good to hear from you! I was hoping to avoid the use of git but that > may be the only way. The version is 21.08.5. I checked the "old" > packages from SchedMD and they begin part way through 2024 so that > won't work. > > I'm very surprised Ubuntu let a package through without a source > package for it. I'm hoping I'm not seeing the tree through the forest > in finding that package. > > Thanks for the help! > > Jeff > > > On Wed, May 15, 2024 at 10:54 AM Lloyd Brown via slurm-users > <slurm-users(a)lists.schedmd.com> wrote: > > Jeff, > > I'm not sure what version is in the Ubuntu packages, as I don't > think they're provided by SchedMD, and I'm having trouble finding > the right one on packages.ubuntu.com > <http://packages.ubuntu.com>. Having said that, SchedMD is pretty > good about using tags in their github repo > (https://github.com/schedmd/slurm), to represent the releases. > For example, the "slurm-23-11-6-1" tag corresponds to release > 23.11.6. It's pretty straightforward to clone the repo, and do > something like "git checkout -b MY_LOCAL_BRANCH_NAME TAG_NAME" to > get the version you're after. > > > Lloyd > > -- > Lloyd Brown > HPC Systems Administrator > Office of Research Computing > Brigham Young University > http://rc.byu.edu > > On 5/15/24 08:35, Jeffrey Layton via slurm-users wrote: >> Good morning, >> >> I have an Ubuntu 22.04 server where I installed Slurm from the >> Ubuntu packages. I now want to install pyxis but it says I need >> the Slurm sources. In Ubuntu 22.04, is there a package that has >> the source code? How to download the sources I need from github? >> >> Thanks! >> >> Jeff > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > -- Lloyd Brown HPC Systems Administrator Office of Research Computing Brigham Young University http://rc.byu.edu

1 0

Job Invalid Account
by Joe Teumer 15 May '24

15 May '24

We installed slurm 23.11.5 and we are receiving "JobId=n has invalid account" for every sbatch job. We are not using the slurm accounting/user database; we are using uniform UIDs and GIDs across the cluster. The jobs run and complete; can these invalid account errors be ignored or silenced? Job Submission Environment: id joteumer uid=938401109(joteumer) gid=938400513(SPG) groups=938400513(SPG),27(sudo) Slurm Worker Node: id joteumer uid=938401109(joteumer) gid=938400513(SPG) groups=938400513(SPG),27(sudo) slurmctld log: [2024-04-18T09:46:40.000] sched: JobId=18 has invalid account scontrol show job 18 JobId=18 JobName=simplejob.sh UserId=joteumer(938401109) GroupId=SPG(938400513) MCS_label=N/A Priority=1 Nice=0 *Account=(null) *QOS=(null) Submit another sbatch job and update the job to include an Account scontrol update jobid=19 Account=joteumer [2024-04-18T09:56:05.126] _slurm_rpc_submit_batch_job: JobId=19 InitPrio=1 usec=485 [2024-04-18T09:56:06.000] sched: JobId=19 has invalid account [2024-04-18T09:56:17.000] debug: set_job_failed_assoc_qos_ptr: Filling in assoc for JobId=19 Assoc=0 [2024-04-18T09:56:17.000] sched: JobId=19 has invalid account [2024-04-18T09:56:17.588] debug: set_job_failed_assoc_qos_ptr: Filling in assoc for JobId=19 Assoc=0 [2024-04-18T09:56:27.505] _slurm_rpc_update_job: complete JobId=19 uid=0 usec=110 [2024-04-18T09:56:28.000] sched: JobId=19 has invalid account scontrol show job 19 JobId=19 JobName=simplejob.sh UserId=joteumer(938401109) GroupId=SPG(938400513) MCS_label=N/A Priority=1 Nice=0 Account=(null) QOS=(null) JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) 19 SPG simplejob joteumer PENDING 0:00 18:00:00 1 (InvalidAccount)

2 1

Slurm Cleaning Up $XDG_RUNTIME_DIR Before It Should?
by Arnuld 15 May '24

15 May '24

I am using the latest slurm. It runs fine for scripts. But if I give it a container then it kills it as soon as I submit the job. Is slurm cleaning up the $XDG_RUNTIME_DIR before it should? This is the log: [2024-05-15T08:00:35.143] [90.0] debug2: _generate_patterns: StepId=90.0 TaskId=-1 [2024-05-15T08:00:35.143] [90.0] debug3: _get_container_state: command argv[0]=/bin/sh [2024-05-15T08:00:35.143] [90.0] debug3: _get_container_state: command argv[1]=-c [2024-05-15T08:00:35.143] [90.0] debug3: _get_container_state: command argv[2]=crun --rootless=true --root=/run/user/1000/ state slurm2.acog.90.0.-1 [2024-05-15T08:00:35.167] [90.0] debug: _get_container_state: RunTimeQuery rc:256 output:error opening file `/run/user/1000/slurm2.acog.90.0.-1/status`: No such file or directory [2024-05-15T08:00:35.167] [90.0] error: _get_container_state: RunTimeQuery failed rc:256 output:error opening file `/run/user/1000/slurm2.acog.90.0.-1/status`: No such file or directory [2024-05-15T08:00:35.167] [90.0] debug: container already dead [2024-05-15T08:00:35.167] [90.0] debug3: _generate_spooldir: task:0 pattern:%m/oci-job%j-%s/task-%t/ path:/var/spool/slurmd/oci-job90-0/task-0/ [2024-05-15T08:00:35.167] [90.0] debug2: _generate_patterns: StepId=90.0 TaskId=0 [2024-05-15T08:00:35.168] [90.0] debug3: _generate_spooldir: task:-1 pattern:%m/oci-job%j-%s/ path:/var/spool/slurmd/oci-job90-0/ [2024-05-15T08:00:35.168] [90.0] stepd_cleanup: done with step (rc[0x100]:Unknown error 256, cleanup_rc[0x0]:No error) [2024-05-15T08:00:35.275] debug3: in the service_connection [2024-05-15T08:00:35.278] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-05-15T08:00:35.278] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-05-15T08:00:35.278] debug: _rpc_terminate_job: uid = 64030 JobId=90 [2024-05-15T08:00:35.278] debug: credential for job 90 revoked

2 2

Slurm release candidate version 24.05.0rc1 available for testing
by Marshall Garey 14 May '24

14 May '24

We are pleased to announce the availability of Slurm release candidate 24.05.0rc1. To highlight some new features coming in 24.05: - (Optional) isolated Job Step management. Enabled on a job-by-job basis with the --stepmgr option, or globally through SlurmctldParameters=enable_stepmgr. - Federation - Allow for client command operation while SlurmDBD is unavailable. - New MaxTRESRunMinsPerAccount and MaxTRESRunMinsPerUser QOS limits. - New USER_DELETE reservation flag. - New Flags=rebootless option on Features for node_features/helpers which indicates the given feature can be enabled without rebooting the node. - Cloud power management options: New "max_powered_nodes=<limit>" option in SlurmctldParamters, and new SuspendExcNodes=<nodes>:<count> syntax allowing for <count> nodes out of a given node list to be excluded. - StdIn/StdOut/StdErr now stored in SlurmDBD accounting records for batch jobs. - New switch/nvidia_imex plugin for IMEX channel management on NVIDIA systems. - New RestrictedCoresPerGPU option at the Node level, designed to ensure GPU workloads always have access to a certain number of CPUs even when nodes are running non-GPU workloads concurrently. This is the first release candidate of the upcoming 24.05 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 24.05.x version and we will address them before the first production 24.05.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Submitting from an untrusted node
by Rike-Benjamin Schuppner 14 May '24

14 May '24

Hi, If I understand it correctly, the MUNGE and SACK authentication modules naturally require that no-one can get access to the key. This means that we should not use our normal workstations to which our users have physical access to run any jobs, nor could our users use the workstations to submit jobs to the compute nodes. They would have to ssh to a specific submit node and only then could they schedule their jobs. Is there an elegant way to enable job submission from any computer (possibly requiring that users type their password for the submit node – or to their ssh key – at some point)? (All computers/users use the same LDAP server for logins.) Best /rike

2 1

Which "oci.conf" to use?
by Arnuld 14 May '24

14 May '24

I have installed slurm and podman. I have replaced podman's default runtime as per the documentation to "slurm". Documentation says I need to choose one oci.conf: https://slurm.schedmd.com/containers.html#example Which one should I use? runc? crun? nvidia?

1 0

any way to allow interactive jobs or ssh in Slurm 23.02 when node is draining?
by Robert Kudyba 13 May '24

13 May '24

We use Bright Cluster Manager with SLurm 23.02 on RHEL9. I know about pam_slurm_adopt https://slurm.schedmd.com/pam_slurm_adopt.html which does not appear to come by default with the Bright 'cm' package of Slurm. Currently ssh to a node gets: Login not allowed: no running jobs and no WLM allocations We have 8 GPUs on a node so when we drain a node, which can have up to a 5 day job, no new jobs can run. And since we have 20+ TB (yes TB) local drives, researchers have their work and files on them to retrieve. Is there a way to use /etc/security/access.conf to work around this at least temporarily until the reboot and then we can revert? Thanks! Rob

2 2

Announcing Slurm-web v3.0.0, open source web dashboard for Slurm
by Rémi Palancher 13 May '24

13 May '24

Hello Slurm users, Some of you may find interest in the new major version of Slurm-web v3.0.0, an open source web dashboard for Slurm: https://slurm-web.com Slurm-web provides a reactive & responsive web interface to track jobs with intuitive insights and advanced visualizations to monitor status of HPC supercomputers in your organization. The software is released under GPLv3 [1]. This new version is based on official Slurm REST API slurmrestd and adopts modern web technologies to provide many features: - Instant jobs filtering and sorting - Live jobs status update - Advanced visualization of node status with racking topology - Intuitive visualization of QOS and advanced reservations - Multi-clusters support - LDAP authentication - Advanced RBAC permissions management - Transparent caching For the next releases, a roadmap is published with many features ideas [2]. Quick start guide to install: http://docs.rackslab.io/slurm-web/install/quickstart.html RPM and deb packages are published for easy installation and upgrade on all most popular Linux distributions. I hope you will like it! [1] https://github.com/rackslab/Slurm-web [2] https://slurm-web.com/roadmap/ -- Rémi Palancher Rackslab: Open Source Solutions for HPC Operations https://rackslab.io

1 0

srun launched mpi job occasionally core dumps
by Henderson, Brent 10 May '24

10 May '24

Greetings Slurm gurus -- I've been having an issue where very occasionally an srun launched OpenMPI job launched will die during startup within MPI_Init(). E.g. srun -N 8 --ntasks-per-node=1 ./hello_world_mpi. Same binary launched with mpirun does not experience the issue. E.g. mpirun -n 64 -H cn01,... ./hello_world_mpi. The failure rate seems to be in the 0.5% - 1.0% range when using srun for launch. SW stack is self-built with: * Dual socket AMD nodes * RHEL 9.3 base system + tools * Single 100 Gb card per host * hwloc 2.9.3 * pmix 4.2.9 (5.0.2 also tried but continued to see the same issues) * slurm 23.11.6 (started with 23.11.5 - update did not change the behavior) * openmpi 5.0.3 The MPI code is a simple hello_world_mpi.c - anything that goes through startup via srun - does not seem to matter. Application core dump looks like the following regardless of the test running: [cn04:1194785] *** Process received signal *** [cn04:1194785] Signal: Segmentation fault (11) [cn04:1194785] Signal code: Address not mapped (1) [cn04:1194785] Failing at address: 0xe0 [cn04:1194785] [ 0] /lib64/libc.so.6(+0x54db0)[0x7f54e6254db0] [cn04:1194785] [ 1] /share/openmpi/5.0.3/lib/libmpi.so.40(mca_pml_ob1_recv_frag_callback_match+0x7d)[0x7f54e67eab3d] [cn04:1194785] [ 2] /share/openmpi/5.0.3/lib/libopen-pal.so.80(+0xa7d8c)[0x7f54e6566d8c] [cn04:1194785] [ 3] /lib64/libevent_core-2.1.so.7(+0x21b88)[0x7f54e649cb88] [cn04:1194785] [ 4] /lib64/libevent_core-2.1.so.7(event_base_loop+0x577)[0x7f54e649e7a7] [cn04:1194785] [ 5] /share/openmpi/5.0.3/lib/libopen-pal.so.80(+0x222af)[0x7f54e64e12af] [cn04:1194785] [ 6] /share/openmpi/5.0.3/lib/libopen-pal.so.80(opal_progress+0x85)[0x7f54e64e1365] [cn04:1194785] [ 7] /share/openmpi/5.0.3/lib/libmpi.so.40(ompi_mpi_init+0x46d)[0x7f54e663ce7d] [cn04:1194785] [ 8] /share/openmpi/5.0.3/lib/libmpi.so.40(MPI_Init+0x5e)[0x7f54e66711ae] [cn04:1194785] [ 9] /home/brent/bin/ior-3.0.1/ior[0x403780] [cn04:1194785] [10] /lib64/libc.so.6(+0x3feb0)[0x7f54e623feb0] [cn04:1194785] [11] /lib64/libc.so.6(__libc_start_main+0x80)[0x7f54e623ff60] [cn04:1194785] [12] /home/brent/bin/ior-3.0.1/ior[0x4069d5] [cn04:1194785] *** End of error message *** More than one rank can die with the same stacktrace on a node when this happens - I've seen as many as 6. One other interesting note is that if I change my srun command line to include strace (e.g. srun -N 8 --ntasks-per-node=8 strace <strace-options> ./hello_world_mpi) the issue appears to go away. 0 failures in ~2500 runs. Another thing that seems to help is to disabling cgroups in the slurm.conf. After the change, saw 0 failures in >6100 hello_world_mpi runs. The changes in the slurm.conf were - original: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup,task/affinity JobAcctGatherType=jobacct_gather/cgroup Changed ProctrackType=proctrack/linuxproc TaskPlugin=task/affinity JobAcctGatherType=jobacct_gather/linux My cgroup.conf file contains: ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes AllowedRamSpace=95 Curious is anyone has any thoughts on next steps to help figure out what might be going on and how to resolve it. Currently, I'm planning to back down to the 23.02.7 release and see how that goes but open to other suggestions. Thanks, Brent

2 4

Issues with MIG after update to 23.11
by Cumer Cristiano 10 May '24

10 May '24

Dear list, after an update form 22.05 to 23.11 a host where I'm using MIG started to discard the mig devices during the NVML setuop. I can see those lines in the log: slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) where the MIG devices are discarded during the start of the slurmd deamon. this is my gres.conf file cat /etc/slurm/gres.conf ################################################################## # Slurm's Generic Resource (GRES) configuration file ################################################################## AutoDetect=nvml and that still works for hosts on 22.05 Any idea of what is going on? I tried to disable MIG on the upgraded server, and then all the A100 are recognized without issues. Thanks Cristiano slurmd: debug: Log file re-opened slurmd: debug2: hwloc_topology_init slurmd: debug2: hwloc_topology_load slurmd: debug2: hwloc_topology_export_xml slurmd: debug: CPUs:64 Boards:1 Sockets:2 CoresPerSocket:32 ThreadsPerCore:1 slurmd: debug: cgroup/v1: init: Cgroup v1 plugin loaded slurmd: debug2: hwloc_topology_init slurmd: debug2: xcpuinfo_hwloc_topo_load: xml file (/var/lib/slurm/slurmd/hwloc_topo_whole.xml) found slurmd: debug: CPUs:64 Boards:1 Sockets:2 CoresPerSocket:32 ThreadsPerCore:1 slurmd: debug: gres/gpu: init: loaded slurmd: debug: gpu/nvml: init: init: GPU NVML plugin loaded slurmd: debug2: gpu/nvml: _nvml_init: Successfully initialized NVML slurmd: debug: gpu/nvml: _get_system_gpu_list_nvml: Systems Graphics Driver Version: 550.54.15 slurmd: debug: gpu/nvml: _get_system_gpu_list_nvml: NVML Library Version: 12.550.54.15 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVML API Version: 11 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Total CPU count: 64 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device count: 4 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 0: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-808805ee-2ae5-ee6d-14fd-6e63028d4a2a slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:1:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:01:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: -1,0,0,0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 24-31 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 24-31 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: enabled slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG count: 2 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 0, MIG index 0: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-ab1b2cb1-d690-5564-885e-056f23c1c618 slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-ab1b2cb1-d690-5564-885e-056f23c1c618 slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 1 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 12 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 13 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia0,/dev/nvidia-caps/nvidia-cap12,/dev/nvidia-caps/nvidia-cap13 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 0, MIG index 1: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-c43ecbcb-9f3f-5972-8eb9-6a2a09848bd5 slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-c43ecbcb-9f3f-5972-8eb9-6a2a09848bd5 slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 2 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 21 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 22 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia0,/dev/nvidia-caps/nvidia-cap21,/dev/nvidia-caps/nvidia-cap22 slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 1: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-05911075-e3e1-973b-708a-33a77ddd381c slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:65:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:41:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,-1,0,0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia1 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 8-15 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 8-15 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: enabled slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG count: 2 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 1, MIG index 0: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-36b02f58-efa6-5071-9956-510c8a8e705a slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-36b02f58-efa6-5071-9956-510c8a8e705a slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 1 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 147 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 148 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia1,/dev/nvidia-caps/nvidia-cap147,/dev/nvidia-caps/nvidia-cap148 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 1, MIG index 1: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-c5ca2779-f34d-544b-af52-55ef14aa0af6 slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-c5ca2779-f34d-544b-af52-55ef14aa0af6 slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 2 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 156 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 157 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia1,/dev/nvidia-caps/nvidia-cap156,/dev/nvidia-caps/nvidia-cap157 slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 2: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-7b67964f-9217-908f-042f-f37f06cbb1a2 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:129:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:81:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,-1,4 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia2 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 56-63 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 56-63 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 3: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-87d7526f-6a29-214e-1c24-722a20211263 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:193:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:C1:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,4,-1 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia3 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 40-47 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 40-47 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: gpu/nvml: _get_system_gpu_list_nvml: 4 GPU system device(s) detected slurmd: debug: Gres GPU plugin: Merging configured GRES with system GPUs slurmd: debug2: gres/gpu: _merge_system_gres_conf: gres_list_conf: slurmd: debug2: GRES[gpu] Type:a100_3g.39gb Count:4 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: debug2: GRES[gpu] Type:a100-sxm4-80gb Count:2 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):56-63 Links:0,0,-1,4 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:(null) slurmd: debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):40-47 Links:0,0,4,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: warning: The following autodetected GPUs are being ignored: slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):24-31 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0,/dev/nvidia-caps/nvidia-cap12,/dev/nvidia-caps/nvidia-cap13 UniqueId:MIG-ab1b2cb1-d690-5564-885e-056f23c1c618 slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):24-31 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0,/dev/nvidia-caps/nvidia-cap21,/dev/nvidia-caps/nvidia-cap22 UniqueId:MIG-c43ecbcb-9f3f-5972-8eb9-6a2a09848bd5 slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):8-15 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1,/dev/nvidia-caps/nvidia-cap147,/dev/nvidia-caps/nvidia-cap148 UniqueId:MIG-36b02f58-efa6-5071-9956-510c8a8e705a slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):8-15 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1,/dev/nvidia-caps/nvidia-cap156,/dev/nvidia-caps/nvidia-cap157 UniqueId:MIG-c5ca2779-f34d-544b-af52-55ef14aa0af6 slurmd: debug2: gres/gpu: _merge_system_gres_conf: gres_list_gpu slurmd: debug2: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):56-63 Links:0,0,-1,4 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:(null) slurmd: debug2: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):40-47 Links:0,0,4,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:(null) slurmd: debug: Gres GPU plugin: Final merged GRES list: slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):56-63 Links:0,0,-1,4 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:(null) slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):40-47 Links:0,0,4,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:(null) slurmd: Gres Name=gpu Type=a100-sxm4-80gb Count=1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=a100-sxm4-80gb Count=1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: topology/default: init: topology Default plugin loaded slurmd: CPU frequency setting not configured for this node slurmd: debug: Resource spec: No specialized cores configured by default on this node slurmd: debug: Resource spec: Reserved system memory limit not configured for this node slurmd: debug: task/cgroup: init: Tasks containment cgroup plugin loaded slurmd: task/affinity: init: task affinity plugin loaded with CPU mask 0xffffffffffffffff slurmd: debug: spank: opening plugin stack /var/lib/slurm/slurmd/conf-cache/plugstack.conf slurmd: debug: /var/lib/slurm/slurmd/conf-cache/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf" slurmd: cred/munge: init: Munge credential signature plugin loaded slurmd: warning: Core limit is only 0 KB slurmd: slurmd version 23.11.4 started slurmd: debug2: No acct_gather.conf file (/var/lib/slurm/slurmd/conf-cache/acct_gather.conf) slurmd: debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded slurmd: debug: MPI: Loading all types slurmd: debug: mpi/pmix_v4: init: PMIx plugin loaded slurmd: debug: mpi/pmix_v4: init: PMIx plugin loaded slurmd: debug2: No mpi.conf file (/var/lib/slurm/slurmd/conf-cache/mpi.conf) slurmd: slurmd started on Fri, 10 May 2024 14:30:30 +0000 slurmd: CPUs=64 Boards=1 Sockets=2 Cores=32 Threads=1 Memory=1031664 TmpDisk=0 Uptime=2512944 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) ^Cslurmd: got shutdown request slurmd: all threads complete slurmd: debug: mpi/pmix_v4: fini: (null) [0]: mpi_pmix.c:203: (null): call fini() slurmd: debug: mpi/pmix_v4: fini: (null) [0]: mpi_pmix.c:203: (null): call fini() slurmd: debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded slurmd: debug2: acct_gather_profile_startpoll: poll already ended! slurmd: debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded slurmd: debug: task/affinity: fini: task affinity plugin unloaded slurmd: debug: hash/k12: fini: fini: unloading KangarooTwelve hash plugin slurmd: debug: gres/gpu: fini: unloading ^Cslurmd: debug2: gpu/nvml: _nvml_shutdown: Successfully shut down NVML slurmd: debug: gpu/nvml: fini: fini: unloading GPU NVML plugin slurmd: debug2: acct_gather_profile_startpoll: poll already ended! slurmd: debug: cgroup/v1: fini: unloading Cgroup v1 plugin slurmd: cred/munge: fini: Munge credential signature plugin unloaded slurmd: Slurmd shutdown completing

1 0

Trouble Running Slurm C Extension Plugin
by Glen MacLachlan 10 May '24

10 May '24

Hi, We have a plugin in Lua that mostly does what we want but there are features available in the C extension that are not available to lua. For that reason, we are attempting to convert to C using the guidance found here: https://slurm.schedmd.com/job_submit_plugins.html#building. We arrived here because the lua plugins don't seem to stretch enough to cover the use case we were looking at, i.e., branching off of the value of alloc_id or, for that matter, get_sid(). The goal is to disallow interactive allocations (i.e., salloc) on specific partitions while allowing it on others. However, we've run into an issue with our C plugin right out of the gate and I've included a minimal reproducer as an example which is basically a "Hello World" type of test (job_submit_disallow_salloc.c, see attached). *Expectation* What we expect to happen is a sort of hello-world result with a message being written to a /tmp/min_repo.log but that does not occur. It seems that the plugin does not get run at all when jobs are submitted. Jobs still run as expected but the plugin seems to be ignored. *Steps* We compile gcc -fPIC -DHAVE_CONFIG_H -I /modules/source/slurm-23.02.4 -g -O2 -pthread -fno-gcse -Werror -Wall -g -O0 -fno-strict-aliasing -MT job_submit_disallow_salloc.lo -MD -MP -MF .deps/job_submit_disallow_salloc.Tpo -c job_submit_disallow_salloc.c -o .libs/job_submit_disallow_salloc.o mv .deps/job_submit_disallow_salloc.Tpo .deps/job_submit_disallow_salloc.Plo and link gcc -shared -fPIC -DPIC .libs/job_submit_disallow_salloc.o -O2 -pthread -O0 -pthread -Wl,-soname -Wl,job_submit_disallow_salloc.so -o job_submit_disallow_salloc.so Check links after copying to /usr/lib64/slurm: ldd /usr/lib64/slurm/job_submit_disallow_salloc.so linux-vdso.so.1 (0x00007ffe467aa000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1c02095000) libc.so.6 => /lib64/libc.so.6 (0x00007f1c01cd0000) /lib64/ld-linux-x86-64.so.2 (0x00007f1c024b7000) Can someone point out what we are doing incorrectly or how we might troubleshoot this issue? Kindest regards, Glen *Reproducer* The minimal reproducer is basically a "hello world" for C extensions which I've pasted below (I've also attached it for convenience): #include <slurm/slurm.h> #include <slurm/slurm_errno.h> #include <stdio.h> #include "src/slurmctld/slurmctld.h" const char plugin_name[] = "Min Reproducer"; const char plugin_type[] = "job_submit/disallow_salloc"; const uint32_t plugin_version = SLURM_VERSION_NUMBER; extern int job_submit(job_desc_msg_t *job_desc, uint32_t submit_uid, char **err_msg) { FILE *fp; fp = fopen("/tmp/min_repo.log", "w"); fprintf(fp,"Hello!"); fclose(fp); return SLURM_SUCCESS; } int job_modify(job_desc_msg_t *job_desc, job_record_t *job_ptr, uint32_t submit_uid, char **err_msg) { return SLURM_SUCCESS; }

3 2

RunTimeQuery never configured in oci.conf
by Arnuld 10 May '24

10 May '24

I am using Slurm integrated with Podman. It runs the container fine and the Controller daemon log always says "WEXITSTATUS 0" . Container also runs successfully (it runs the python test program with no errors). But there are two things that I noticed: - slurmd.log says: " error: _get_container_state: RunTimeQuery failed rc:256 output:RunTimeQuery never configured in oci.conf" - podman --log-level=debug reports: "[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied" How can I diagnose further to know what's wrong

1 0

GPU GRES verification and some really broad questions.
by Shooktija S N 10 May '24

10 May '24

Hi, I am a complete slurm-admin and sys-admin noob trying to set up a 3 node Slurm cluster. I have managed to get a minimum working example running, in which I am able to use a GPU (NVIDIA GeForce RTX 4070 ti) as a GRES. This is *slurm.conf* without the comment lines: root@server1:/etc/slurm# grep -v "#" slurm.confClusterName=DlabClusterSlurmctldHost=server1GresTypes=gpuProctrackType=proctrack/linuxprocReturnToService=1SlurmctldPidFile=/var/run/slurmctld.pidSlurmctldPort=6817SlurmdPidFile=/var/run/slurmd.pidSlurmdPort=6818SlurmdSpoolDir=/var/spool/slurmdSlurmUser=rootStateSaveLocation=/var/spool/slurmctldTaskPlugin=task/affinity,task/cgroupInactiveLimit=0KillWait=30MinJobAge=300SlurmctldTimeout=120SlurmdTimeout=300Waittime=0SchedulerType=sched/backfillSelectType=select/cons_tresJobCompType=jobcomp/noneJobAcctGatherFrequency=30SlurmctldDebug=infoSlurmctldLogFile=/var/log/slurmctld.logSlurmdDebug=debug3SlurmdLogFile=/var/log/slurmd.logNodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP This is *gres.conf* (only one line), each node has been assigned its corresponding NodeName: root@server1:/etc/slurm# cat gres.confNodeName=server1 Name=gpu File=/dev/nvidia0 Those are the only config files I have. I have a few general questions, loosely arranged in ascending order of generality: 1) I have enabled the allocation of GPU resources as a GRES and have tested this by running: shookti@server1:~$ srun --nodes=3 --gpus=3 --label hostname2: server30: server11: server2 Is this a good way to check if the configs have worked correctly? How else can I easily check if the GPU GRES has been properly configured? 2) I want to reserve a few CPU cores, and a few gigs of memory for use by non slurm related tasks. According to the documentation, I am to use CoreSpecCount <https://slurm.schedmd.com/slurm.conf.html#OPT_CoreSpecCount> and MemSpecLimit <https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit> to achieve this. The documentation for CoreSpecCount says "the Slurm daemon slurmd may either be confined to these resources (the default) or prevented from using these resources", how do I change this default behaviour to have the config specify the cores reserved for non slurm stuff instead of specifying how many cores slurm can use? 3) While looking up examples online on how to run Python scripts inside a conda env, I have seen that the line 'module load conda' should be run before running 'conda activate myEnv' in the sbatch submission script. The command 'module' did not exist until I installed the apt package 'environment-modules', but now I see that conda is not listed as a module that can be loaded when I check using the command 'module avail'. How do I fix this? 4) A very broad question: while managing the resources being used by a program, slurm might happen to split the resources across multiple computers that might not necessarily have the files required by this program to run. For example, a python script that requires the package 'numpy' to function but that package was not installed on all of the computers. How are such things dealt with? Is the module approach meant to fix this problem? In my previous question, if I had a python script that users usually run just by running a command like 'python3 someScript.py' instead of running it within a conda environment, how should I enable slurm to manage the resources required by this script? Would I have to install all the packages required by this script on all the computers that are in the cluster? 5) Related to the previous question: I have set up my 3 nodes in such a way that all the users' home directories are stored on a ceph cluster <https://en.wikipedia.org/wiki/Ceph_(software)> created using the hard drives from all the 3 nodes, which essentially means that a user's home directory is mounted at the same location on all 3 computers - making a user's data visible to all 3 nodes. Does this make the process of managing the dependencies of a program as described in the previous question easier? I realise that programs having to read and write to files on the hard drives of a ceph cluster is not really the fastest so I am planning on having users use the /tmp/ directory for speed critical reading and writing, as the OSs have been installed on NVME drives.

2 1

Problems with gres.conf
by Gestió Servidors 10 May '24

10 May '24

Hello, I am trying to rewrite my gres.conf file. Before changes, this file was just like this: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 # you can seee that nodes node-gpu-1 and node-gpu-2 have two GPUs each one, whereas nodes node-gpu-3 and node-gpu-4 have only one GPU each one And my slurmd.conf was this: [...] AccountingStorageTRES=gres/gpu GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 [...] With this configuration, all seems works fine, except slurmctld.log reports: [...] error: _node_config_validate: gres/gpu: invalid GRES core specification (0-11) on node node-gpu-3 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-1 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-2 error: _node_config_validate: gres/gpu: invalid GRES core specification (0-7) on node node-gpu-4 [...] However, even these errors, users can submit jobs and request GPUs resources. Now, I have tried to reconfigure gres.conf and slurmd.conf in this way: gres.conf: Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 # there is no NodeName attribute slurmd.conf: [...] NodeName=node-gpu-1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 # there is no CPUs attribute [...] With this new configuration, nodes with GPU start correctly slurmd.service daemon, but nodes without GPU (node-worker-[0-22]) can't start slurmd.service daemon and returns this error: [...] error: Waiting for gres.conf file /dev/nvidia0 fatal: can't stat gres.conf file /dev/nvidia0: No such file or directory [...] It seems SLURM is waiting that "node-workers" have also an nvidia GPU but not, theses nodes haven't GPU... So, where is my configuration error? I have read in https://slurm.schedmd.com/gres.conf.html about syntax and examples but it seems I'm doing some wrong. Thanks!!

1 0

Re: srun launched mpi job occasionally core dumps
by Henderson, Brent 08 May '24

08 May '24

Thanks for the suggestion Ole - I'll see if I can get that in the mix to try over the next few days. I can report that 23.02.7 tree had the same issues, so going backwards on the slurm bits did not have any impact. Brent

1 0

Partition Preemption Configuration Question
by Jason Simms 08 May '24

08 May '24

Hello all, The Slurm docs have me a bit confused... I'm wanting to enable job preemption on certain partitions but not others. I *presume* I would set PreemptType=preempt/partition_prio globally, but then on the partitions where I don't want jobs to be able to be preempted, I would set PreemptMode=off within the configuration for that specific partition. The documentation, however, says that setting PreemptMode=off at a partition level "is only compatible with PreemptType=preempt/none at a global level" yet then immediately says that doing so is a "common use case for this parameter is to set it on a partition to disable preemption for that partition," which indicates preemption would still be allowable for other partitions. If PreemptType is set to preempt/none globally, and I *cannot* set that as an option for a given partition (at least, the documentation doesn't indicate that is a valid parameter for a partition), wouldn't preemption be disabled globally anyway? The wording seems odd to me and almost contradictory. Is it possible to have PreemptType=preempt/partition_prio set globally, yet also disable it on specific partitions with PreemptMode=off? Is PreemptType actually a valid configuration option for specific partitions? Thanks for any guidance. Warmest regards, Jason -- *Jason L. Simms, Ph.D., M.P.H.* Manager of Research Computing Swarthmore College Information Technology Services (610) 328-8102 Schedule a meeting: https://calendly.com/jlsimms

3 3

Container Jobs "hanging"
by sean+slurm＠superorbital.io 08 May '24

08 May '24

Hello. I am new to this list and Slurm overall. I have a lot of experience in computer operations, including Kubernetes, but I am currently exploring Slurm in some depth. I have set up a small cluster and, in general, have gotten things working, but when I try to run a container job, it runs the command but then appears to hang as if the job container is still running. So, running the following works, but it never returns to the prompt unless I use [Control-C]. $ srun --container /shared_fs/shared/oci_images/alpine uptime 19:21:47 up 20:43, 0 users, load average: 0.03, 0.25, 0.15 I'm unsure if something is misconfigured or if I'm misunderstanding how this should work, but any help and/or pointers would be greatly appreciated. Thanks! Sean

1 0

2025

2024

slurm-users ----- 2025 ----- April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users