- slurm-users - lists.schedmd.com

Fwd: Re: Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs.
by Hongyi Zhao 28 May '24

28 May '24

---------- Forwarded message --------- From: Hermann Schwärzler <hermann.schwaerzler(a)uibk.ac.at> Date: Tue, May 28, 2024 at 4:10 PM Subject: Re: [slurm-users] Re: Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs. To: Hongyi Zhao <hongyi.zhao(a)gmail.com> Hi Zhao, On 5/28/24 03:08, Hongyi Zhao wrote: [...] > > What's the complete content of cli_filter.lua and where should I put this file? [...] Below you find the complete content of our cli_filter.lua. It has to be put into the same directory as "slurm.conf". --------------------------------- 8< --------------------------------- -- see https://github.com/SchedMD/slurm/blob/master/etc/cli_filter.lua.example function slurm_cli_pre_submit(options, pack_offset) return slurm.SUCCESS end function slurm_cli_setup_defaults(options, early_pass) -- Make --hint=nomultithread the default behavior -- if users specify an other --hint=XX option then -- it will override the setting done here options['hint'] = 'nomultithread' return slurm.SUCCESS end function slurm_cli_post_submit(offset, job_id, step_id) return slurm.SUCCESS end --------------------------------- >8 --------------------------------- Hopefully this helps... Regards, Hermann -- Assoc. Prof. Hongsheng Zhao <hongyi.zhao(a)gmail.com> Theory and Simulation of Materials Hebei Vocational University of Technology and Engineering No. 473, Quannan West Street, Xindu District, Xingtai, Hebei province

1 0

Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs.
by Hongyi Zhao 27 May '24

27 May '24

Dear Slurm Users, I am experiencing a significant performance discrepancy when running the same VASP job through the Slurm scheduler compared to running it directly with mpirun. I am hoping for some insights or advice on how to resolve this issue. System Information: Slurm Version: 21.08.5 OS: Ubuntu 22.04.4 LTS (Jammy) Job Submission Script: #!/usr/bin/env bash #SBATCH -N 1 #SBATCH -D . #SBATCH --output=%j.out #SBATCH --error=%j.err ##SBATCH --time=2-00:00:00 #SBATCH --ntasks=36 #SBATCH --mem=64G echo '#######################################################' echo "date = $(date)" echo "hostname = $(hostname -s)" echo "pwd = $(pwd)" echo "sbatch = $(which sbatch | xargs realpath -e)" echo "" echo "WORK_DIR = $WORK_DIR" echo "SLURM_SUBMIT_DIR = $SLURM_SUBMIT_DIR" echo "SLURM_JOB_NUM_NODES = $SLURM_JOB_NUM_NODES" echo "SLURM_NTASKS = $SLURM_NTASKS" echo "SLURM_NTASKS_PER_NODE = $SLURM_NTASKS_PER_NODE" echo "SLURM_CPUS_PER_TASK = $SLURM_CPUS_PER_TASK" echo "SLURM_JOBID = $SLURM_JOBID" echo "SLURM_JOB_NODELIST = $SLURM_JOB_NODELIST" echo "SLURM_NNODES = $SLURM_NNODES" echo "SLURMTMPDIR = $SLURMTMPDIR" echo '#######################################################' echo "" module purge > /dev/null 2>&1 module load vasp ulimit -s unlimited mpirun vasp_std Performance Observation: When running the job through Slurm: werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ grep LOOP OUTCAR LOOP: cpu time 14.4893: real time 14.5049 LOOP: cpu time 14.3538: real time 14.3621 LOOP: cpu time 14.3870: real time 14.3568 LOOP: cpu time 15.9722: real time 15.9018 LOOP: cpu time 16.4527: real time 16.4370 LOOP: cpu time 16.7918: real time 16.7781 LOOP: cpu time 16.9797: real time 16.9961 LOOP: cpu time 15.9762: real time 16.0124 LOOP: cpu time 16.8835: real time 16.9008 LOOP: cpu time 15.2828: real time 15.2921 LOOP+: cpu time 176.0917: real time 176.0755 When running the job directly with mpirun: werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ mpirun -n 36 vasp_std werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ grep LOOP OUTCAR LOOP: cpu time 9.0072: real time 9.0074 LOOP: cpu time 9.0515: real time 9.0524 LOOP: cpu time 9.1896: real time 9.1907 LOOP: cpu time 10.1467: real time 10.1479 LOOP: cpu time 10.2691: real time 10.2705 LOOP: cpu time 10.4330: real time 10.4340 LOOP: cpu time 10.9049: real time 10.9055 LOOP: cpu time 9.9718: real time 9.9714 LOOP: cpu time 10.4511: real time 10.4470 LOOP: cpu time 9.4621: real time 9.4584 LOOP+: cpu time 110.0790: real time 110.0739 Could you provide any insights or suggestions on what might be causing this performance issue? Are there any specific configurations or settings in Slurm that I should check or adjust to align the performance more closely with the direct mpirun execution? Thank you for your time and assistance. Best regards, Zhao -- Assoc. Prof. Hongsheng Zhao <hongyi.zhao(a)gmail.com> Theory and Simulation of Materials Hebei Vocational University of Technology and Engineering No. 473, Quannan West Street, Xindu District, Xingtai, Hebei province

4 10

Building Slurm debian package vs building from source
by Arnuld 24 May '24

24 May '24

We have several nodes, most of which have different Linux distributions (distro for short). Controller has a different distro as well. The only common thing between controller and all the does is that all of them ar x86_64. I can install Slurm using package manager on all the machines but this will not work because controller will have a different version of Slurm compared to the nodes (21.08 vs 23.11) If I build from source then I see two solutions: - build a deb package - build a custom package (./configure, make, make install) Building a debian package on the controller and then distributing the binaries on nodes won't work either because that binary will start looking for the shared libraries that it was built for and those don't exist on the nodes. So the only solution I have is to build a static binary using a custom package. Am I correct or is there another solution here?

4 7

Slurm Build Error
by Arnuld 23 May '24

23 May '24

Getting this error when I run "make install": echo >>"lib_ref.lo" /bin/bash ../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o lib_ref.la lib_ref.lo -lpthread -lm -lresolv libtool: link: ar cr .libs/lib_ref.a libtool: link: ranlib .libs/lib_ref.a libtool: link: ( cd ".libs" && rm -f "lib_ref.la" && ln -s "../lib_ref.la" " lib_ref.la" ) /bin/bash ../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -o sacctmgr account_functions.o archive_functions.o association_functions.o config_functions.o cluster_functions.o common.o event_functions.o federation_functions.o file_functions.o instance_functions.o runaway_job_functions.o job_functions.o reservation_functions.o resource_functions.o sacctmgr.o qos_functions.o txn_functions.o user_functions.o wckey_functions.o problem_functions.o tres_function.o -Wl,-rpath=/root/slurm-slurm-23-11-7-1/z/lib/slurm -L../../src/api/.libs -lslurmfull -export-dynamic -lreadline -lhistory lib_ref.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o sacctmgr account_functions.o archive_functions.o association_functions.o config_functions.o cluster_functions.o common.o event_functions.o federation_functions.o file_functions.o instance_functions.o runaway_job_functions.o job_functions.o reservation_functions.o resource_functions.o sacctmgr.o qos_functions.o txn_functions.o user_functions.o wckey_functions.o problem_functions.o tres_function.o -Wl,-rpath=/root/slurm-slurm-23-11-7-1/z/lib/slurm -Wl,--export-dynamic -L../../src/api/.libs /root/slurm-slurm-23-11-7-1/src/api/.libs/libslurmfull.a -lreadline -lhistory ./.libs/lib_ref.a -lpthread -lm -lresolv -pthread /usr/bin/ld: sacctmgr.o: warning: relocation against `_binary_usage_txt_end' in read-only section `.text' /usr/bin/ld: sacctmgr.o: in function `_usage': /root/slurm-slurm-23-11-7-1/src/sacctmgr/sacctmgr.c:926:(.text+0x1f): undefined reference to `_binary_usage_txt_start' /usr/bin/ld: /root/slurm-slurm-23-11-7-1/src/sacctmgr/sacctmgr.c:926:(.text+0x26): undefined reference to `_binary_usage_txt_end' /usr/bin/ld: warning: creating DT_TEXTREL in a PIE collect2: error: ld returned 1 exit status make[3]: *** [Makefile:672: sacctmgr] Error 1 make[3]: Leaving directory '/root/slurm-slurm-23-11-7-1/src/sacctmgr' make[2]: *** [Makefile:520: all-recursive] Error 1 make[2]: Leaving directory '/root/slurm-slurm-23-11-7-1/src' make[1]: *** [Makefile:621: all-recursive] Error 1 make[1]: Leaving directory '/root/slurm-slurm-23-11-7-1' make: *** [Makefile:520: all] Error 2 ----------------------------------------------------------------------------------------- I used these config options: ./configure --enable-debug --enable-pam --disable-sview --disable-shared --with-munge --with-json --with-yaml --with-http-parser --with-pmix --with-lz4 --with-hwloc --with-jwt --with-libcurl --with-freeipmi --with-rdkafka --with-bpf --prefix=/root/slurm-slurm-23-11-7-1/z

1 0

memory high water mark reporting
by Emyr James 22 May '24

22 May '24

Hi, We are trying out slurm having been running grid engine for a long while. In grid engine, the cgroups peak memory and max_rss are generated at the end of a job and recorded. It logs the information from the cgroup hierarchy as well as doing a getrusage call right at the end on the parent pid of the whole job "container" before cleaning up. With slurm it seems that the only way memory is recorded is by the acct gather polling. I am trying to add something in an epilog script to get the memory.peak but It looks like the cgroup hierarchy has been destroyed by the time the epilog is run. Where in the code is the cgroup hierarchy cleared up ? Is there no way to add something in so that the accounting is updated during the job cleanup process so that peak memory usage can be accurately logged ? I can reduce the polling interval from 30s to 5s but don't know if this causes a lot of overhead and in any case this seems to not be a sensible way to get values that should just be determined right at the end by an event rather than using polling. Many thanks, Emyr

5 10

Slurm version 23.11.7 is now available
by Marshall Garey 21 May '24

21 May '24

We are pleased to announce the availability of Slurm version 23.11.7. The 23.11.7 release fixes a few potential crashes in slurmctld when using less common options on job submission, slurmrestd compatibility with auth/slurm, and some additional minor and moderate severity bugs. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Marshall > -- slurmrestd - Correct OpenAPI specification for > 'GET /slurm/v0.0.40/jobs/state' having response as null. > -- Allow running jobs on overlapping partitions if jobs don't specify -s. > -- Fix segfault when requesting a shared gres along with an exclusive > allocation. > -- Fix regression in 23.02 where afternotok and afterok dependencies were > rejected for federated jobs not running on the origin cluster of the > submitting job. > -- slurmctld - Disable job table locking while job state cache is active when > replying to `squeue --only-job-state` or `GET /slurm/v0.0.40/jobs/state`. > -- Fix sanity check when setting tres-per-task on the job allocation as well as > the step. > -- slurmrestd - Fix compatiblity with auth/slurm. > -- Fix issue where TRESRunMins gets off correct value if using > QOS UsageFactor != 1. > -- slurmrestd - Require `user` and `association_condition` fields to be > populated for requests to 'POST /slurmdb/v0.0.40/users_association'. > -- Avoid a slurmctld crash with extra_constraints enabled when a job requests > certain invalid --extra values. > -- `scancel --ctld` and `DELETE /slurm/v0.0/40/jobs` - Fix support for job > array expressions (e.g. 1_[3-5]). Also fix signaling a single pending array > task (e.g. 1_10), which previously signaled the whole array job instead. > -- Fix a possible slurmctld segfault when at some point we failed to create an > external launcher step. > -- Allow the slurmctld to open a connection to the slurmdbd if the first > attempt fails due to a protocol error. > -- mpi/cray_shasta - Fix launch for non-het-steps within a hetjob. > -- sacct - Fix "gpuutil" TRES usage output being incorrect when using --units. > -- Fix a rare deadlock on slurmctld shutdown or reconfigure. > -- Fix issue that only left one thread on each core available when "CPUs=" is > configured to total thread count on multi-threaded hardware and no other > topology info ("Sockets=", "CoresPerSocket", etc.) is configured. > -- Fix the external launcher step not being allocated a VNI when requested. > -- jobcomp/kafka - Fix payload length when producing and sending a message. > -- scrun - Avoid a crash if RunTimeDelete is called before the container > finishes. > -- Save the slurmd's cred_state while reconfiguring to prevent the loss job > credentials.

1 0

SLUG Call for Papers Deadline
by Victoria Hobson 21 May '24

21 May '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information and a high-level schedule can be found here:https://slug24.splashthat.com/ The deadline to submit a presentation abstract is Friday, May 31st. We do not intend to extend this deadline. If you are interested in presenting your own usage, developments, site report, tutorial, etc about Slurm, please fill out the following form:https://forms.gle/N7bFo5EzwuTuKkBN7 Notifications of final presentations accepted will go out by Friday, June 14th. -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

Slurm not allocating correct cgroup cpu ids in srun step (possible bug)
by Ashley Wright 21 May '24

21 May '24

Hi, At our site we have recently upgraded to Slurm 23.11.5 and are having trouble with MPI jobs doing srun inside a sbatch'ed script. The cgroup does not appear to be setup correctly for the srun (step_0). As an example $ cat /sys/fs/cgroup/cpuset/slurm/uid_11000..../job..../cpuset.cpus 0,2-3,68-69,96,98-99,164-165 $ cat /sys/fs/cgroup/cpuset/slurm/uid_11000..../job..../step_0/cpuset.cpus 0,2,68,96,98,164 The sbatch is allocated a range of cpus in the cgroup. However, when step_0 is run, only some of those CPUs are in the group. I have noticed that it is always the range which is missing, ie 2-5 only 2 is included, 3,4,5 are missing. This also only happens if there are multiple groups of cpus in the allocations. ie only 1-12 would be fine, however 1-12,15-20 would result in 1,15 only. The sbatch also seems fine, with step_batch and step_extern being allocated correctly. This causes numerous issues with MPI jobs as they end up overloading cpus. We are running our nodes with threading enabled on the CPUs, and with cgroups and affinity plugins. I have attached our slurm.conf to show our settings. Our /etc/slurm/cgroup.conf is ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes We have turned on logging at debug2 level, but I haven't yet found anything useful. Happy for a suggestion on what to look for. Is anyone able to provide any advice on where to go next to try and identify the issue? Regards, Ashley Wright

1 0

Running slurm on alternate ports
by Alan Stange 20 May '24

20 May '24

Hello all, for testing purposes, we would like to run slurm on ports different from the default values. No problems in setting this up. But how does one tell srun/sbatch/etc what the different port numbers are? I see no command line options to specify a port or an alternate configuration file. Thank you, Alan

2 2

Invalid/incorrect gres.conf syntax
by Gestió Servidors 20 May '24

20 May '24

Hello, I have configured my "gres.conf" in this way: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 node-gpu-1 and node-gpu-2 are two systems with two sockets; node-gpu-3 and node-gpu-4 have only one socket. In my "slurm.conf" I have these lines: AccountingStorageTRES=gres/gpu SelectType=select/cons_tres GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 However, slurmctld reports warning logs about "error syntax in Cores attribute" in gres.conf. Where is the syntax error? Thanks a lot!

1 0

Apply an specific QoS to all users that belongs to an specific account
by Gestió Servidors 20 May '24

20 May '24

Hi, I would like to know if it is possible to apply an specific QoS to all users that belongs to an specific account. For example, I have created some new users "user_XX" and, also, I have created their new accounts in SLURM with "sacctmgr create account name=Test" and "sacctmgr create user name=user_XX DefaultAccount=Test". After it, I have changed default QoS "normal" to a new QoS "minimal" (with some limits) to account "Test" (sacctmgr modify account where name=Test set qos=minimal), but what I have seen is that users "user_XX" that belongs to "Test" account continue in QoS "normal" (QoS by default), so it seems that users have not inherited QoS applied to their account. Is there any way to do this and am I doing something wrong? Thanks.

1 0

srun weirdness
by Dj Merrill 17 May '24

17 May '24

I'm running into a strange issue and I'm hoping another set of brains looking at this might help. I would appreciate any feedback. I have two Slurm Clusters. The first cluster is running Slurm 21.08.8 on Rocky Linux 8.9 machines. The second cluster is running Slurm 23.11.6 on Rocky Linux 9.4 machines. This works perfectly fine on the first cluster: $ srun --mem=32G --pty /bin/bash srun: job 93911 queued and waiting for resources srun: job 93911 has been allocated resources and on the resulting shell on the compute node: $ /mnt/local/ollama/ollama help and the ollama help message appears as expected. However, on the second cluster: $ srun --mem=32G --pty /bin/bash srun: job 3 queued and waiting for resources srun: job 3 has been allocated resources and on the resulting shell on the compute node: $ /mnt/local/ollama/ollama help fatal error: failed to reserve page summary memory runtime stack: runtime.throw({0x1240c66?, 0x154fa39a1008?}) runtime/panic.go:1023 +0x5c fp=0x7ffe6be32648 sp=0x7ffe6be32618 pc=0x4605dc runtime.(*pageAlloc).sysInit(0x127b47e8, 0xf8?) runtime/mpagealloc_64bit.go:81 +0x11c fp=0x7ffe6be326b8 sp=0x7ffe6be32648 pc=0x456b7c runtime.(*pageAlloc).init(0x127b47e8, 0x127b47e0, 0x128d88f8, 0x0) runtime/mpagealloc.go:320 +0x85 fp=0x7ffe6be326e8 sp=0x7ffe6be326b8 pc=0x454565 runtime.(*mheap).init(0x127b47e0) runtime/mheap.go:769 +0x165 fp=0x7ffe6be32720 sp=0x7ffe6be326e8 pc=0x451885 runtime.mallocinit() runtime/malloc.go:454 +0xd7 fp=0x7ffe6be32758 sp=0x7ffe6be32720 pc=0x434f97 runtime.schedinit() runtime/proc.go:785 +0xb7 fp=0x7ffe6be327d0 sp=0x7ffe6be32758 pc=0x464397 runtime.rt0_go() runtime/asm_amd64.s:349 +0x11c fp=0x7ffe6be327d8 sp=0x7ffe6be327d0 pc=0x49421c If I ssh directly to the same node on that second cluster (skipping Slurm entirely), and run the same "/mnt/local/ollama/ollama help" command, it works perfectly fine. My first thought was that it might be related to cgroups. I switched the second cluster from cgroups v2 to v1 and tried again, no difference. I tried disabling cgroups on the second cluster by removing all cgroups references in the slurm.conf file but that also made no difference. My guess is something changed with regards to srun between these two Slurm versions, but I'm not sure what. Any thoughts on what might be happening and/or a way to get this to work on the second cluster? Essentially I need a way to request an interactive shell through Slurm that is associated with the requested resources. Should we be using something other than srun for this? Thank you, -Dj

6 12

Slurm DB upgrade failure behavior
by Yuengling, Philip J. 17 May '24

17 May '24

Hi everyone, I'm writing up some Ansible code to manage Slurm software updates, and I haven't found any documentation about slurmdbd behavior if the mysql/mariadb database doesn't upgrade successfully. What I do know is that if it is sucessful I can expect to see "Conversion done: success!" in the slurmdbd log. This is good, but minor updates do not update the database as far as I know. If the Slurm database cannot upgrade upon an update, does it always shut down with a fatal error? What other behaviors should I look for if there is a failure? Cheers, Phil Y

2 1

Removing safely a node
by Ratnasamy, Fritz 17 May '24

17 May '24

Hi, What is the "official" process to remove nodes safely? I have drained the nodes so jobs are completed and put them in down state after they are completely drained. I edited the slurm.conf file to remove the nodes. After some time, I can see that the nodes were removed from the partition with the command sinfo However, I was told I might need to restart the service slurmctld, do you know if it is necessary? Should I also run scontrol reconfig? Best, *Fritz Ratnasamy* Data Scientist Information Technology

3 2

Best practice for jobs resuming from suspended state
by Paul Jones 17 May '24

17 May '24

Hi, We use PreemptMode and PriorityTier within Slurm to suspend low priority jobs when more urgent work needs to be done. This generally works well, but on occasion resumed jobs fail to restart - which is to say Slurm sets the job status to running but the actual code doesn't recover from being suspended. Technically everything is working as expected, but I wondered if there was any best practice to pass onto users about how to cope with this state? Obviously not a direct Slurm question, but wondered if others had experience with this and any advice on how best to limit the impact? Thanks, Paul --

2 1

Location of Slurm source packages?
by Jeffrey Layton 15 May '24

15 May '24

Good morning, I have an Ubuntu 22.04 server where I installed Slurm from the Ubuntu packages. I now want to install pyxis but it says I need the Slurm sources. In Ubuntu 22.04, is there a package that has the source code? How to download the sources I need from github? Thanks! Jeff

2 2

Re: Location of Slurm source packages?
by Lloyd Brown 15 May '24

15 May '24

Jeff, I'm not sure what version is in the Ubuntu packages, as I don't think they're provided by SchedMD, and I'm having trouble finding the right one on packages.ubuntu.com. Having said that, SchedMD is pretty good about using tags in their github repo (https://github.com/schedmd/slurm), to represent the releases. For example, the "slurm-23-11-6-1" tag corresponds to release 23.11.6. It's pretty straightforward to clone the repo, and do something like "git checkout -b MY_LOCAL_BRANCH_NAME TAG_NAME" to get the version you're after. Lloyd -- Lloyd Brown HPC Systems Administrator Office of Research Computing Brigham Young University http://rc.byu.edu On 5/15/24 08:35, Jeffrey Layton via slurm-users wrote: > Good morning, > > I have an Ubuntu 22.04 server where I installed Slurm from the Ubuntu > packages. I now want to install pyxis but it says I need the Slurm > sources. In Ubuntu 22.04, is there a package that has the source code? > How to download the sources I need from github? > > Thanks! > > Jeff

3 3

Re: Location of Slurm source packages?
by Lloyd Brown 15 May '24

15 May '24

Jeff, Dang. That's really old. I'm not sure I would run one that old, to be honest. Too many missing security fixes and added features. It's never been that hard to do a 'git clone' and the normal configure/make/make install process with slurm. Someone else made me aware of this, in case it's easier: https://slurm.schedmd.com/quickstart_admin.html#debuild Lloyd On 5/15/24 08:57, Jeffrey Layton wrote: > Lloyd, > > Good to hear from you! I was hoping to avoid the use of git but that > may be the only way. The version is 21.08.5. I checked the "old" > packages from SchedMD and they begin part way through 2024 so that > won't work. > > I'm very surprised Ubuntu let a package through without a source > package for it. I'm hoping I'm not seeing the tree through the forest > in finding that package. > > Thanks for the help! > > Jeff > > > On Wed, May 15, 2024 at 10:54 AM Lloyd Brown via slurm-users > <slurm-users(a)lists.schedmd.com> wrote: > > Jeff, > > I'm not sure what version is in the Ubuntu packages, as I don't > think they're provided by SchedMD, and I'm having trouble finding > the right one on packages.ubuntu.com > <http://packages.ubuntu.com>. Having said that, SchedMD is pretty > good about using tags in their github repo > (https://github.com/schedmd/slurm), to represent the releases. > For example, the "slurm-23-11-6-1" tag corresponds to release > 23.11.6. It's pretty straightforward to clone the repo, and do > something like "git checkout -b MY_LOCAL_BRANCH_NAME TAG_NAME" to > get the version you're after. > > > Lloyd > > -- > Lloyd Brown > HPC Systems Administrator > Office of Research Computing > Brigham Young University > http://rc.byu.edu > > On 5/15/24 08:35, Jeffrey Layton via slurm-users wrote: >> Good morning, >> >> I have an Ubuntu 22.04 server where I installed Slurm from the >> Ubuntu packages. I now want to install pyxis but it says I need >> the Slurm sources. In Ubuntu 22.04, is there a package that has >> the source code? How to download the sources I need from github? >> >> Thanks! >> >> Jeff > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > -- Lloyd Brown HPC Systems Administrator Office of Research Computing Brigham Young University http://rc.byu.edu

1 0

Job Invalid Account
by Joe Teumer 15 May '24

15 May '24

We installed slurm 23.11.5 and we are receiving "JobId=n has invalid account" for every sbatch job. We are not using the slurm accounting/user database; we are using uniform UIDs and GIDs across the cluster. The jobs run and complete; can these invalid account errors be ignored or silenced? Job Submission Environment: id joteumer uid=938401109(joteumer) gid=938400513(SPG) groups=938400513(SPG),27(sudo) Slurm Worker Node: id joteumer uid=938401109(joteumer) gid=938400513(SPG) groups=938400513(SPG),27(sudo) slurmctld log: [2024-04-18T09:46:40.000] sched: JobId=18 has invalid account scontrol show job 18 JobId=18 JobName=simplejob.sh UserId=joteumer(938401109) GroupId=SPG(938400513) MCS_label=N/A Priority=1 Nice=0 *Account=(null) *QOS=(null) Submit another sbatch job and update the job to include an Account scontrol update jobid=19 Account=joteumer [2024-04-18T09:56:05.126] _slurm_rpc_submit_batch_job: JobId=19 InitPrio=1 usec=485 [2024-04-18T09:56:06.000] sched: JobId=19 has invalid account [2024-04-18T09:56:17.000] debug: set_job_failed_assoc_qos_ptr: Filling in assoc for JobId=19 Assoc=0 [2024-04-18T09:56:17.000] sched: JobId=19 has invalid account [2024-04-18T09:56:17.588] debug: set_job_failed_assoc_qos_ptr: Filling in assoc for JobId=19 Assoc=0 [2024-04-18T09:56:27.505] _slurm_rpc_update_job: complete JobId=19 uid=0 usec=110 [2024-04-18T09:56:28.000] sched: JobId=19 has invalid account scontrol show job 19 JobId=19 JobName=simplejob.sh UserId=joteumer(938401109) GroupId=SPG(938400513) MCS_label=N/A Priority=1 Nice=0 Account=(null) QOS=(null) JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) 19 SPG simplejob joteumer PENDING 0:00 18:00:00 1 (InvalidAccount)

2 1

Slurm Cleaning Up $XDG_RUNTIME_DIR Before It Should?
by Arnuld 15 May '24

15 May '24

I am using the latest slurm. It runs fine for scripts. But if I give it a container then it kills it as soon as I submit the job. Is slurm cleaning up the $XDG_RUNTIME_DIR before it should? This is the log: [2024-05-15T08:00:35.143] [90.0] debug2: _generate_patterns: StepId=90.0 TaskId=-1 [2024-05-15T08:00:35.143] [90.0] debug3: _get_container_state: command argv[0]=/bin/sh [2024-05-15T08:00:35.143] [90.0] debug3: _get_container_state: command argv[1]=-c [2024-05-15T08:00:35.143] [90.0] debug3: _get_container_state: command argv[2]=crun --rootless=true --root=/run/user/1000/ state slurm2.acog.90.0.-1 [2024-05-15T08:00:35.167] [90.0] debug: _get_container_state: RunTimeQuery rc:256 output:error opening file `/run/user/1000/slurm2.acog.90.0.-1/status`: No such file or directory [2024-05-15T08:00:35.167] [90.0] error: _get_container_state: RunTimeQuery failed rc:256 output:error opening file `/run/user/1000/slurm2.acog.90.0.-1/status`: No such file or directory [2024-05-15T08:00:35.167] [90.0] debug: container already dead [2024-05-15T08:00:35.167] [90.0] debug3: _generate_spooldir: task:0 pattern:%m/oci-job%j-%s/task-%t/ path:/var/spool/slurmd/oci-job90-0/task-0/ [2024-05-15T08:00:35.167] [90.0] debug2: _generate_patterns: StepId=90.0 TaskId=0 [2024-05-15T08:00:35.168] [90.0] debug3: _generate_spooldir: task:-1 pattern:%m/oci-job%j-%s/ path:/var/spool/slurmd/oci-job90-0/ [2024-05-15T08:00:35.168] [90.0] stepd_cleanup: done with step (rc[0x100]:Unknown error 256, cleanup_rc[0x0]:No error) [2024-05-15T08:00:35.275] debug3: in the service_connection [2024-05-15T08:00:35.278] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-05-15T08:00:35.278] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-05-15T08:00:35.278] debug: _rpc_terminate_job: uid = 64030 JobId=90 [2024-05-15T08:00:35.278] debug: credential for job 90 revoked

2 2

Slurm release candidate version 24.05.0rc1 available for testing
by Marshall Garey 14 May '24

14 May '24

We are pleased to announce the availability of Slurm release candidate 24.05.0rc1. To highlight some new features coming in 24.05: - (Optional) isolated Job Step management. Enabled on a job-by-job basis with the --stepmgr option, or globally through SlurmctldParameters=enable_stepmgr. - Federation - Allow for client command operation while SlurmDBD is unavailable. - New MaxTRESRunMinsPerAccount and MaxTRESRunMinsPerUser QOS limits. - New USER_DELETE reservation flag. - New Flags=rebootless option on Features for node_features/helpers which indicates the given feature can be enabled without rebooting the node. - Cloud power management options: New "max_powered_nodes=<limit>" option in SlurmctldParamters, and new SuspendExcNodes=<nodes>:<count> syntax allowing for <count> nodes out of a given node list to be excluded. - StdIn/StdOut/StdErr now stored in SlurmDBD accounting records for batch jobs. - New switch/nvidia_imex plugin for IMEX channel management on NVIDIA systems. - New RestrictedCoresPerGPU option at the Node level, designed to ensure GPU workloads always have access to a certain number of CPUs even when nodes are running non-GPU workloads concurrently. This is the first release candidate of the upcoming 24.05 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 24.05.x version and we will address them before the first production 24.05.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Submitting from an untrusted node
by Rike-Benjamin Schuppner 14 May '24

14 May '24

Hi, If I understand it correctly, the MUNGE and SACK authentication modules naturally require that no-one can get access to the key. This means that we should not use our normal workstations to which our users have physical access to run any jobs, nor could our users use the workstations to submit jobs to the compute nodes. They would have to ssh to a specific submit node and only then could they schedule their jobs. Is there an elegant way to enable job submission from any computer (possibly requiring that users type their password for the submit node – or to their ssh key – at some point)? (All computers/users use the same LDAP server for logins.) Best /rike

2 1

Which "oci.conf" to use?
by Arnuld 14 May '24

14 May '24

I have installed slurm and podman. I have replaced podman's default runtime as per the documentation to "slurm". Documentation says I need to choose one oci.conf: https://slurm.schedmd.com/containers.html#example Which one should I use? runc? crun? nvidia?

1 0

any way to allow interactive jobs or ssh in Slurm 23.02 when node is draining?
by Robert Kudyba 13 May '24

13 May '24

We use Bright Cluster Manager with SLurm 23.02 on RHEL9. I know about pam_slurm_adopt https://slurm.schedmd.com/pam_slurm_adopt.html which does not appear to come by default with the Bright 'cm' package of Slurm. Currently ssh to a node gets: Login not allowed: no running jobs and no WLM allocations We have 8 GPUs on a node so when we drain a node, which can have up to a 5 day job, no new jobs can run. And since we have 20+ TB (yes TB) local drives, researchers have their work and files on them to retrieve. Is there a way to use /etc/security/access.conf to work around this at least temporarily until the reboot and then we can revert? Thanks! Rob

2 2

Announcing Slurm-web v3.0.0, open source web dashboard for Slurm
by Rémi Palancher 13 May '24

13 May '24

Hello Slurm users, Some of you may find interest in the new major version of Slurm-web v3.0.0, an open source web dashboard for Slurm: https://slurm-web.com Slurm-web provides a reactive & responsive web interface to track jobs with intuitive insights and advanced visualizations to monitor status of HPC supercomputers in your organization. The software is released under GPLv3 [1]. This new version is based on official Slurm REST API slurmrestd and adopts modern web technologies to provide many features: - Instant jobs filtering and sorting - Live jobs status update - Advanced visualization of node status with racking topology - Intuitive visualization of QOS and advanced reservations - Multi-clusters support - LDAP authentication - Advanced RBAC permissions management - Transparent caching For the next releases, a roadmap is published with many features ideas [2]. Quick start guide to install: http://docs.rackslab.io/slurm-web/install/quickstart.html RPM and deb packages are published for easy installation and upgrade on all most popular Linux distributions. I hope you will like it! [1] https://github.com/rackslab/Slurm-web [2] https://slurm-web.com/roadmap/ -- Rémi Palancher Rackslab: Open Source Solutions for HPC Operations https://rackslab.io

1 0

srun launched mpi job occasionally core dumps
by Henderson, Brent 10 May '24

10 May '24

Greetings Slurm gurus -- I've been having an issue where very occasionally an srun launched OpenMPI job launched will die during startup within MPI_Init(). E.g. srun -N 8 --ntasks-per-node=1 ./hello_world_mpi. Same binary launched with mpirun does not experience the issue. E.g. mpirun -n 64 -H cn01,... ./hello_world_mpi. The failure rate seems to be in the 0.5% - 1.0% range when using srun for launch. SW stack is self-built with: * Dual socket AMD nodes * RHEL 9.3 base system + tools * Single 100 Gb card per host * hwloc 2.9.3 * pmix 4.2.9 (5.0.2 also tried but continued to see the same issues) * slurm 23.11.6 (started with 23.11.5 - update did not change the behavior) * openmpi 5.0.3 The MPI code is a simple hello_world_mpi.c - anything that goes through startup via srun - does not seem to matter. Application core dump looks like the following regardless of the test running: [cn04:1194785] *** Process received signal *** [cn04:1194785] Signal: Segmentation fault (11) [cn04:1194785] Signal code: Address not mapped (1) [cn04:1194785] Failing at address: 0xe0 [cn04:1194785] [ 0] /lib64/libc.so.6(+0x54db0)[0x7f54e6254db0] [cn04:1194785] [ 1] /share/openmpi/5.0.3/lib/libmpi.so.40(mca_pml_ob1_recv_frag_callback_match+0x7d)[0x7f54e67eab3d] [cn04:1194785] [ 2] /share/openmpi/5.0.3/lib/libopen-pal.so.80(+0xa7d8c)[0x7f54e6566d8c] [cn04:1194785] [ 3] /lib64/libevent_core-2.1.so.7(+0x21b88)[0x7f54e649cb88] [cn04:1194785] [ 4] /lib64/libevent_core-2.1.so.7(event_base_loop+0x577)[0x7f54e649e7a7] [cn04:1194785] [ 5] /share/openmpi/5.0.3/lib/libopen-pal.so.80(+0x222af)[0x7f54e64e12af] [cn04:1194785] [ 6] /share/openmpi/5.0.3/lib/libopen-pal.so.80(opal_progress+0x85)[0x7f54e64e1365] [cn04:1194785] [ 7] /share/openmpi/5.0.3/lib/libmpi.so.40(ompi_mpi_init+0x46d)[0x7f54e663ce7d] [cn04:1194785] [ 8] /share/openmpi/5.0.3/lib/libmpi.so.40(MPI_Init+0x5e)[0x7f54e66711ae] [cn04:1194785] [ 9] /home/brent/bin/ior-3.0.1/ior[0x403780] [cn04:1194785] [10] /lib64/libc.so.6(+0x3feb0)[0x7f54e623feb0] [cn04:1194785] [11] /lib64/libc.so.6(__libc_start_main+0x80)[0x7f54e623ff60] [cn04:1194785] [12] /home/brent/bin/ior-3.0.1/ior[0x4069d5] [cn04:1194785] *** End of error message *** More than one rank can die with the same stacktrace on a node when this happens - I've seen as many as 6. One other interesting note is that if I change my srun command line to include strace (e.g. srun -N 8 --ntasks-per-node=8 strace <strace-options> ./hello_world_mpi) the issue appears to go away. 0 failures in ~2500 runs. Another thing that seems to help is to disabling cgroups in the slurm.conf. After the change, saw 0 failures in >6100 hello_world_mpi runs. The changes in the slurm.conf were - original: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup,task/affinity JobAcctGatherType=jobacct_gather/cgroup Changed ProctrackType=proctrack/linuxproc TaskPlugin=task/affinity JobAcctGatherType=jobacct_gather/linux My cgroup.conf file contains: ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes AllowedRamSpace=95 Curious is anyone has any thoughts on next steps to help figure out what might be going on and how to resolve it. Currently, I'm planning to back down to the 23.02.7 release and see how that goes but open to other suggestions. Thanks, Brent

2 4

Issues with MIG after update to 23.11
by Cumer Cristiano 10 May '24

10 May '24

Dear list, after an update form 22.05 to 23.11 a host where I'm using MIG started to discard the mig devices during the NVML setuop. I can see those lines in the log: slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) where the MIG devices are discarded during the start of the slurmd deamon. this is my gres.conf file cat /etc/slurm/gres.conf ################################################################## # Slurm's Generic Resource (GRES) configuration file ################################################################## AutoDetect=nvml and that still works for hosts on 22.05 Any idea of what is going on? I tried to disable MIG on the upgraded server, and then all the A100 are recognized without issues. Thanks Cristiano slurmd: debug: Log file re-opened slurmd: debug2: hwloc_topology_init slurmd: debug2: hwloc_topology_load slurmd: debug2: hwloc_topology_export_xml slurmd: debug: CPUs:64 Boards:1 Sockets:2 CoresPerSocket:32 ThreadsPerCore:1 slurmd: debug: cgroup/v1: init: Cgroup v1 plugin loaded slurmd: debug2: hwloc_topology_init slurmd: debug2: xcpuinfo_hwloc_topo_load: xml file (/var/lib/slurm/slurmd/hwloc_topo_whole.xml) found slurmd: debug: CPUs:64 Boards:1 Sockets:2 CoresPerSocket:32 ThreadsPerCore:1 slurmd: debug: gres/gpu: init: loaded slurmd: debug: gpu/nvml: init: init: GPU NVML plugin loaded slurmd: debug2: gpu/nvml: _nvml_init: Successfully initialized NVML slurmd: debug: gpu/nvml: _get_system_gpu_list_nvml: Systems Graphics Driver Version: 550.54.15 slurmd: debug: gpu/nvml: _get_system_gpu_list_nvml: NVML Library Version: 12.550.54.15 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVML API Version: 11 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Total CPU count: 64 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device count: 4 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 0: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-808805ee-2ae5-ee6d-14fd-6e63028d4a2a slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:1:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:01:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: -1,0,0,0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 24-31 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 24-31 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: enabled slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG count: 2 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 0, MIG index 0: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-ab1b2cb1-d690-5564-885e-056f23c1c618 slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-ab1b2cb1-d690-5564-885e-056f23c1c618 slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 1 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 12 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 13 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia0,/dev/nvidia-caps/nvidia-cap12,/dev/nvidia-caps/nvidia-cap13 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 0, MIG index 1: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-c43ecbcb-9f3f-5972-8eb9-6a2a09848bd5 slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-c43ecbcb-9f3f-5972-8eb9-6a2a09848bd5 slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 2 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 21 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 22 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia0,/dev/nvidia-caps/nvidia-cap21,/dev/nvidia-caps/nvidia-cap22 slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 1: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-05911075-e3e1-973b-708a-33a77ddd381c slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:65:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:41:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,-1,0,0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia1 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 8-15 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 8-15 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: enabled slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG count: 2 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 1, MIG index 0: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-36b02f58-efa6-5071-9956-510c8a8e705a slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-36b02f58-efa6-5071-9956-510c8a8e705a slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 1 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 147 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 148 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia1,/dev/nvidia-caps/nvidia-cap147,/dev/nvidia-caps/nvidia-cap148 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 1, MIG index 1: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-c5ca2779-f34d-544b-af52-55ef14aa0af6 slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-c5ca2779-f34d-544b-af52-55ef14aa0af6 slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 2 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 156 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 157 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia1,/dev/nvidia-caps/nvidia-cap156,/dev/nvidia-caps/nvidia-cap157 slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 2: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-7b67964f-9217-908f-042f-f37f06cbb1a2 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:129:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:81:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,-1,4 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia2 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 56-63 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 56-63 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 3: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-87d7526f-6a29-214e-1c24-722a20211263 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:193:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:C1:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,4,-1 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia3 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 40-47 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 40-47 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: gpu/nvml: _get_system_gpu_list_nvml: 4 GPU system device(s) detected slurmd: debug: Gres GPU plugin: Merging configured GRES with system GPUs slurmd: debug2: gres/gpu: _merge_system_gres_conf: gres_list_conf: slurmd: debug2: GRES[gpu] Type:a100_3g.39gb Count:4 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: debug2: GRES[gpu] Type:a100-sxm4-80gb Count:2 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):56-63 Links:0,0,-1,4 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:(null) slurmd: debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):40-47 Links:0,0,4,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: warning: The following autodetected GPUs are being ignored: slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):24-31 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0,/dev/nvidia-caps/nvidia-cap12,/dev/nvidia-caps/nvidia-cap13 UniqueId:MIG-ab1b2cb1-d690-5564-885e-056f23c1c618 slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):24-31 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0,/dev/nvidia-caps/nvidia-cap21,/dev/nvidia-caps/nvidia-cap22 UniqueId:MIG-c43ecbcb-9f3f-5972-8eb9-6a2a09848bd5 slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):8-15 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1,/dev/nvidia-caps/nvidia-cap147,/dev/nvidia-caps/nvidia-cap148 UniqueId:MIG-36b02f58-efa6-5071-9956-510c8a8e705a slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):8-15 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1,/dev/nvidia-caps/nvidia-cap156,/dev/nvidia-caps/nvidia-cap157 UniqueId:MIG-c5ca2779-f34d-544b-af52-55ef14aa0af6 slurmd: debug2: gres/gpu: _merge_system_gres_conf: gres_list_gpu slurmd: debug2: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):56-63 Links:0,0,-1,4 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:(null) slurmd: debug2: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):40-47 Links:0,0,4,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:(null) slurmd: debug: Gres GPU plugin: Final merged GRES list: slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):56-63 Links:0,0,-1,4 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:(null) slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):40-47 Links:0,0,4,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:(null) slurmd: Gres Name=gpu Type=a100-sxm4-80gb Count=1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=a100-sxm4-80gb Count=1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: topology/default: init: topology Default plugin loaded slurmd: CPU frequency setting not configured for this node slurmd: debug: Resource spec: No specialized cores configured by default on this node slurmd: debug: Resource spec: Reserved system memory limit not configured for this node slurmd: debug: task/cgroup: init: Tasks containment cgroup plugin loaded slurmd: task/affinity: init: task affinity plugin loaded with CPU mask 0xffffffffffffffff slurmd: debug: spank: opening plugin stack /var/lib/slurm/slurmd/conf-cache/plugstack.conf slurmd: debug: /var/lib/slurm/slurmd/conf-cache/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf" slurmd: cred/munge: init: Munge credential signature plugin loaded slurmd: warning: Core limit is only 0 KB slurmd: slurmd version 23.11.4 started slurmd: debug2: No acct_gather.conf file (/var/lib/slurm/slurmd/conf-cache/acct_gather.conf) slurmd: debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded slurmd: debug: MPI: Loading all types slurmd: debug: mpi/pmix_v4: init: PMIx plugin loaded slurmd: debug: mpi/pmix_v4: init: PMIx plugin loaded slurmd: debug2: No mpi.conf file (/var/lib/slurm/slurmd/conf-cache/mpi.conf) slurmd: slurmd started on Fri, 10 May 2024 14:30:30 +0000 slurmd: CPUs=64 Boards=1 Sockets=2 Cores=32 Threads=1 Memory=1031664 TmpDisk=0 Uptime=2512944 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) ^Cslurmd: got shutdown request slurmd: all threads complete slurmd: debug: mpi/pmix_v4: fini: (null) [0]: mpi_pmix.c:203: (null): call fini() slurmd: debug: mpi/pmix_v4: fini: (null) [0]: mpi_pmix.c:203: (null): call fini() slurmd: debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded slurmd: debug2: acct_gather_profile_startpoll: poll already ended! slurmd: debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded slurmd: debug: task/affinity: fini: task affinity plugin unloaded slurmd: debug: hash/k12: fini: fini: unloading KangarooTwelve hash plugin slurmd: debug: gres/gpu: fini: unloading ^Cslurmd: debug2: gpu/nvml: _nvml_shutdown: Successfully shut down NVML slurmd: debug: gpu/nvml: fini: fini: unloading GPU NVML plugin slurmd: debug2: acct_gather_profile_startpoll: poll already ended! slurmd: debug: cgroup/v1: fini: unloading Cgroup v1 plugin slurmd: cred/munge: fini: Munge credential signature plugin unloaded slurmd: Slurmd shutdown completing

1 0

Trouble Running Slurm C Extension Plugin
by Glen MacLachlan 10 May '24

10 May '24

Hi, We have a plugin in Lua that mostly does what we want but there are features available in the C extension that are not available to lua. For that reason, we are attempting to convert to C using the guidance found here: https://slurm.schedmd.com/job_submit_plugins.html#building. We arrived here because the lua plugins don't seem to stretch enough to cover the use case we were looking at, i.e., branching off of the value of alloc_id or, for that matter, get_sid(). The goal is to disallow interactive allocations (i.e., salloc) on specific partitions while allowing it on others. However, we've run into an issue with our C plugin right out of the gate and I've included a minimal reproducer as an example which is basically a "Hello World" type of test (job_submit_disallow_salloc.c, see attached). *Expectation* What we expect to happen is a sort of hello-world result with a message being written to a /tmp/min_repo.log but that does not occur. It seems that the plugin does not get run at all when jobs are submitted. Jobs still run as expected but the plugin seems to be ignored. *Steps* We compile gcc -fPIC -DHAVE_CONFIG_H -I /modules/source/slurm-23.02.4 -g -O2 -pthread -fno-gcse -Werror -Wall -g -O0 -fno-strict-aliasing -MT job_submit_disallow_salloc.lo -MD -MP -MF .deps/job_submit_disallow_salloc.Tpo -c job_submit_disallow_salloc.c -o .libs/job_submit_disallow_salloc.o mv .deps/job_submit_disallow_salloc.Tpo .deps/job_submit_disallow_salloc.Plo and link gcc -shared -fPIC -DPIC .libs/job_submit_disallow_salloc.o -O2 -pthread -O0 -pthread -Wl,-soname -Wl,job_submit_disallow_salloc.so -o job_submit_disallow_salloc.so Check links after copying to /usr/lib64/slurm: ldd /usr/lib64/slurm/job_submit_disallow_salloc.so linux-vdso.so.1 (0x00007ffe467aa000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1c02095000) libc.so.6 => /lib64/libc.so.6 (0x00007f1c01cd0000) /lib64/ld-linux-x86-64.so.2 (0x00007f1c024b7000) Can someone point out what we are doing incorrectly or how we might troubleshoot this issue? Kindest regards, Glen *Reproducer* The minimal reproducer is basically a "hello world" for C extensions which I've pasted below (I've also attached it for convenience): #include <slurm/slurm.h> #include <slurm/slurm_errno.h> #include <stdio.h> #include "src/slurmctld/slurmctld.h" const char plugin_name[] = "Min Reproducer"; const char plugin_type[] = "job_submit/disallow_salloc"; const uint32_t plugin_version = SLURM_VERSION_NUMBER; extern int job_submit(job_desc_msg_t *job_desc, uint32_t submit_uid, char **err_msg) { FILE *fp; fp = fopen("/tmp/min_repo.log", "w"); fprintf(fp,"Hello!"); fclose(fp); return SLURM_SUCCESS; } int job_modify(job_desc_msg_t *job_desc, job_record_t *job_ptr, uint32_t submit_uid, char **err_msg) { return SLURM_SUCCESS; }

3 2

RunTimeQuery never configured in oci.conf
by Arnuld 10 May '24

10 May '24

I am using Slurm integrated with Podman. It runs the container fine and the Controller daemon log always says "WEXITSTATUS 0" . Container also runs successfully (it runs the python test program with no errors). But there are two things that I noticed: - slurmd.log says: " error: _get_container_state: RunTimeQuery failed rc:256 output:RunTimeQuery never configured in oci.conf" - podman --log-level=debug reports: "[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied" How can I diagnose further to know what's wrong

1 0

GPU GRES verification and some really broad questions.
by Shooktija S N 10 May '24

10 May '24

Hi, I am a complete slurm-admin and sys-admin noob trying to set up a 3 node Slurm cluster. I have managed to get a minimum working example running, in which I am able to use a GPU (NVIDIA GeForce RTX 4070 ti) as a GRES. This is *slurm.conf* without the comment lines: root@server1:/etc/slurm# grep -v "#" slurm.confClusterName=DlabClusterSlurmctldHost=server1GresTypes=gpuProctrackType=proctrack/linuxprocReturnToService=1SlurmctldPidFile=/var/run/slurmctld.pidSlurmctldPort=6817SlurmdPidFile=/var/run/slurmd.pidSlurmdPort=6818SlurmdSpoolDir=/var/spool/slurmdSlurmUser=rootStateSaveLocation=/var/spool/slurmctldTaskPlugin=task/affinity,task/cgroupInactiveLimit=0KillWait=30MinJobAge=300SlurmctldTimeout=120SlurmdTimeout=300Waittime=0SchedulerType=sched/backfillSelectType=select/cons_tresJobCompType=jobcomp/noneJobAcctGatherFrequency=30SlurmctldDebug=infoSlurmctldLogFile=/var/log/slurmctld.logSlurmdDebug=debug3SlurmdLogFile=/var/log/slurmd.logNodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP This is *gres.conf* (only one line), each node has been assigned its corresponding NodeName: root@server1:/etc/slurm# cat gres.confNodeName=server1 Name=gpu File=/dev/nvidia0 Those are the only config files I have. I have a few general questions, loosely arranged in ascending order of generality: 1) I have enabled the allocation of GPU resources as a GRES and have tested this by running: shookti@server1:~$ srun --nodes=3 --gpus=3 --label hostname2: server30: server11: server2 Is this a good way to check if the configs have worked correctly? How else can I easily check if the GPU GRES has been properly configured? 2) I want to reserve a few CPU cores, and a few gigs of memory for use by non slurm related tasks. According to the documentation, I am to use CoreSpecCount <https://slurm.schedmd.com/slurm.conf.html#OPT_CoreSpecCount> and MemSpecLimit <https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit> to achieve this. The documentation for CoreSpecCount says "the Slurm daemon slurmd may either be confined to these resources (the default) or prevented from using these resources", how do I change this default behaviour to have the config specify the cores reserved for non slurm stuff instead of specifying how many cores slurm can use? 3) While looking up examples online on how to run Python scripts inside a conda env, I have seen that the line 'module load conda' should be run before running 'conda activate myEnv' in the sbatch submission script. The command 'module' did not exist until I installed the apt package 'environment-modules', but now I see that conda is not listed as a module that can be loaded when I check using the command 'module avail'. How do I fix this? 4) A very broad question: while managing the resources being used by a program, slurm might happen to split the resources across multiple computers that might not necessarily have the files required by this program to run. For example, a python script that requires the package 'numpy' to function but that package was not installed on all of the computers. How are such things dealt with? Is the module approach meant to fix this problem? In my previous question, if I had a python script that users usually run just by running a command like 'python3 someScript.py' instead of running it within a conda environment, how should I enable slurm to manage the resources required by this script? Would I have to install all the packages required by this script on all the computers that are in the cluster? 5) Related to the previous question: I have set up my 3 nodes in such a way that all the users' home directories are stored on a ceph cluster <https://en.wikipedia.org/wiki/Ceph_(software)> created using the hard drives from all the 3 nodes, which essentially means that a user's home directory is mounted at the same location on all 3 computers - making a user's data visible to all 3 nodes. Does this make the process of managing the dependencies of a program as described in the previous question easier? I realise that programs having to read and write to files on the hard drives of a ceph cluster is not really the fastest so I am planning on having users use the /tmp/ directory for speed critical reading and writing, as the OSs have been installed on NVME drives.

2 1

Problems with gres.conf
by Gestió Servidors 10 May '24

10 May '24

Hello, I am trying to rewrite my gres.conf file. Before changes, this file was just like this: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 # you can seee that nodes node-gpu-1 and node-gpu-2 have two GPUs each one, whereas nodes node-gpu-3 and node-gpu-4 have only one GPU each one And my slurmd.conf was this: [...] AccountingStorageTRES=gres/gpu GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 [...] With this configuration, all seems works fine, except slurmctld.log reports: [...] error: _node_config_validate: gres/gpu: invalid GRES core specification (0-11) on node node-gpu-3 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-1 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-2 error: _node_config_validate: gres/gpu: invalid GRES core specification (0-7) on node node-gpu-4 [...] However, even these errors, users can submit jobs and request GPUs resources. Now, I have tried to reconfigure gres.conf and slurmd.conf in this way: gres.conf: Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 # there is no NodeName attribute slurmd.conf: [...] NodeName=node-gpu-1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 # there is no CPUs attribute [...] With this new configuration, nodes with GPU start correctly slurmd.service daemon, but nodes without GPU (node-worker-[0-22]) can't start slurmd.service daemon and returns this error: [...] error: Waiting for gres.conf file /dev/nvidia0 fatal: can't stat gres.conf file /dev/nvidia0: No such file or directory [...] It seems SLURM is waiting that "node-workers" have also an nvidia GPU but not, theses nodes haven't GPU... So, where is my configuration error? I have read in https://slurm.schedmd.com/gres.conf.html about syntax and examples but it seems I'm doing some wrong. Thanks!!

1 0

Re: srun launched mpi job occasionally core dumps
by Henderson, Brent 08 May '24

08 May '24

Thanks for the suggestion Ole - I'll see if I can get that in the mix to try over the next few days. I can report that 23.02.7 tree had the same issues, so going backwards on the slurm bits did not have any impact. Brent

1 0

Partition Preemption Configuration Question
by Jason Simms 08 May '24

08 May '24

Hello all, The Slurm docs have me a bit confused... I'm wanting to enable job preemption on certain partitions but not others. I *presume* I would set PreemptType=preempt/partition_prio globally, but then on the partitions where I don't want jobs to be able to be preempted, I would set PreemptMode=off within the configuration for that specific partition. The documentation, however, says that setting PreemptMode=off at a partition level "is only compatible with PreemptType=preempt/none at a global level" yet then immediately says that doing so is a "common use case for this parameter is to set it on a partition to disable preemption for that partition," which indicates preemption would still be allowable for other partitions. If PreemptType is set to preempt/none globally, and I *cannot* set that as an option for a given partition (at least, the documentation doesn't indicate that is a valid parameter for a partition), wouldn't preemption be disabled globally anyway? The wording seems odd to me and almost contradictory. Is it possible to have PreemptType=preempt/partition_prio set globally, yet also disable it on specific partitions with PreemptMode=off? Is PreemptType actually a valid configuration option for specific partitions? Thanks for any guidance. Warmest regards, Jason -- *Jason L. Simms, Ph.D., M.P.H.* Manager of Research Computing Swarthmore College Information Technology Services (610) 328-8102 Schedule a meeting: https://calendly.com/jlsimms

3 3

Container Jobs "hanging"
by sean+slurm＠superorbital.io 08 May '24

08 May '24

Hello. I am new to this list and Slurm overall. I have a lot of experience in computer operations, including Kubernetes, but I am currently exploring Slurm in some depth. I have set up a small cluster and, in general, have gotten things working, but when I try to run a container job, it runs the command but then appears to hang as if the job container is still running. So, running the following works, but it never returns to the prompt unless I use [Control-C]. $ srun --container /shared_fs/shared/oci_images/alpine uptime 19:21:47 up 20:43, 0 users, load average: 0.03, 0.25, 0.15 I'm unsure if something is misconfigured or if I'm misunderstanding how this should work, but any help and/or pointers would be greatly appreciated. Thanks! Sean

1 0

scrontab question
by Sandor 08 May '24

08 May '24

I am working out the details of scrontab. My initial testing is giving me an unsolvable question Within scrontab editor I have the following example from the slurm documentation: 0,5,10,15,20,25,30,35,40,45,50,55 * * * * /directory/subdirectory/crontest.sh When I save it, scrontab marks the line with #BAD: I do not understand why. The only difference I have is the directory structure. Is there an underlying assumption that traditional Linux crontab is available to the general user?

4 3

Slurm With Podman - No child processes error
by ARNULD 08 May '24

08 May '24

I have integrated Podman with Slurm as per the docs ( https://slurm.schedmd.com/containers.html#podman-scrun) and when I do a test run: "podman run hello-world" (this runs fine) ---- $ podman run alpine hostname executable file `/usr/bin/hostname` not found in $PATH: No such file or directory srun: error: slurm1: task 0: Exited with exit code 1 ----- $ podman run alpine printenv SLURM_JOB_ID executable file `/usr/bin/printenv` not found in $PATH: No such file or directory srun: error: slurm1: task 0: Exited with exit code 1 scrun: error: run_command_waitpid_timeout: waitpid(67537): No child processes --- podman run alpine uptime 11:31:28 up 5:32, 0 users, load average: 0.00, 0.00, 0.00 scrun: error: run_command_waitpid_timeout: waitpid(68160): No child processes ------ I built a small image from python:alpine3.19 which just prints "hello world" and numbers from 1 to 10. Here is a run: $ podman run -it --rm hello-python $ podman run -it --rm hello-python Hello, world! Numbers from 1 to 10: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] ---- No error with my image. Also I tested podman on another machine without Slurm. Podman with its default runtime prints the hostname fine with "podman run alpine hostname". So something to do with its integration with Slurm. What can I do to diagnose the problem?

1 0

StateSaveLocation and Slurm HA
by Pierre Abele 07 May '24

07 May '24

Hi all, I am looking for a clean way to set up Slurms native high availability feature. I am managing a Slurm cluster with one control node (hosting both slurmctld and slurmdbd), one login node and a few dozen compute nodes. I have a virtual machine that I want to set up as a backup control node. The Slurm documentation says the following about the StateSaveLocation directory: > The directory used should be on a low-latency local disk to prevent file system delays from affecting Slurm performance. If using a backup host, the StateSaveLocation should reside on a file system shared by the two hosts. We do not recommend using NFS to make the directory accessible to both hosts, but do recommend a shared mount that is accessible to the two controllers and allows low-latency reads and writes to the disk. If a controller comes up without access to the state information, queued and running jobs will be cancelled. [1] My question: How do I implement the shared file system for the StateSaveLocation? I do not want to introduce a single point of failure by having a single node that hosts the StateSaveLocation, neither do I want to put that directory on the clusters NFS storage since outages/downtime of the storage system will happen at some point and I do not want that to cause an outage of the Slurm controller. Any help or ideas would be appreciated. Best, Pierre [1] https://slurm.schedmd.com/quickstart_admin.html#Config -- Pierre Abele, M.Sc. HPC Administrator Max-Planck-Institute for Evolutionary Anthropology Department of Primate Behavior and Evolution Deutscher Platz 6 04103 Leipzig Room: U2.80 E-Mail: pierre_abele(a)eva.mpg.de Phone: +49 (0) 341 3550 245

3 2

"token expired" errors with auth/slurm
by Fabio Ranalli 07 May '24

07 May '24

Hi there, We've updated to 23.11.6 and replaced MUNGE with SACK. Performance and stability have both been pretty good, but we're occasionally seeing this in the slurmctld.log /[2024-05-07T03:50:16.638] error: decode_jwt: token expired at 1715053769 [2024-05-07T03:50:16.638] error: cred_p_unpack: decode_jwt() failed [2024-05-07T03:50:16.638] error: Malformed RPC of type REQUEST_BATCH_JOB_LAUNCH(4005) received [2024-05-07T03:50:16.641] error: slurm_receive_msg_and_forward: [[headnode.internal]:58286] failed: Header lengths are longer than data received [2024-05-07T03:50:16.648] error: service_connection: slurm_receive_msg: Header lengths are longer than data received/ it seems to impact a subset of nodes: jobs get killed and no new ones are allocated. Full functionality can be restored by simply restarting slurmctld first, and then slurmd. Is the token expected to actually expire? I didn't see this possibility mentioned in the docs. The problem occurs on an R&D cloud cluster based on EL9, with a pretty "flat" setup. _headnode_: configless slurmctld, slurmdbd, mariadb, nfsd _elastic compute nodes_: autofs, slurmd *//etc/slurm/slurm.conf/* AuthType=auth/slurm AuthInfo=use_client_ids CredType=cred/slurm *//etc/slurm/slurmdbd.conf/* AuthType=auth/slurm AuthInfo=use_client_ids Has anyone else encountered the same error? Thanks, Fabio -- *Fabio Ranalli* | Principal Systems Administrator Schrödinger, Inc. <https://schrodinger.com>

1 0

FreeBSD/aarch64: ld: error: unknown emulation: elf_aarch64
by Nuno Teixeira 06 May '24

06 May '24

Hello, I'm new maintainer of slurm FreeBSD port <https://www.freshports.org/sysutils/slurm-wlm/> and for a long time it failing on aarch64 platform. Any clues? > ld: error: unknown emulation: elf_aarch64 `clang -print-targets`: Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V thumb - Thumb thumbeb - Thumb (big endian) x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 Full log: https://pkg-status.freebsd.org/ampere2/data/main-arm64-default/pb78025a96ed… Thanks, -- Nuno Teixeira FreeBSD UNIX: <eduardo(a)FreeBSD.org> Web: https://FreeBSD.org

3 7

Rootless Docker Errors with Slurm
by ARNULD 06 May '24

06 May '24

I am trying to integrate Rootless Docker with Slurm. have set-up Rootless Docker as per the docs "https://slurm.schedmd.com/containers.html" . I have scrum.lua, oci.conf (for crun) and slurm.conf in place. Then "~/.config/docker/daemon.json" and "~/.config/systemd/user/docker.service.d/override.conf" are in place too. But I can't seem to get it to work: $ docker run $DOCKER_SECURITY alpine /bin/printenv SLURM_JOB_ID Unable to find image 'alpine:latest' locally latest: Pulling from library/alpine 4abcf2066143: Pull complete Digest: sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b Status: Downloaded newer image for alpine:latest docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/user/1000/docker-exec/containerd/daemon/io.containerd.runtime.v2.task/moby/97e8dd767977ac03ab7af54c015c0fd5dfd26e737771b977acb7e41f799023aa/log.json: no such file or directory): /usr/bin/scrun did not terminate successfully: exit status 1: unknown. One thing is if I don't use Slurm as the runtime for Docker (if I remove the " ~/.config/docker/daemon.json") then docker runs fine

1 0

Invalid/incorrect gres.conf syntax
by Gestió Servidors 06 May '24

06 May '24

Hello, I have configured my "gres.conf" in this way: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 node-gpu-1 and node-gpu-2 are two systems with two sockets; node-gpu-3 and node-gpu-4 have only one socket. In my "slurm.conf" I have these lines: AccountingStorageTRES=gres/gpu SelectType=select/cons_tres GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 Thanks a lot!

1 0

slurm reservation: how to use more nodes (bigger than NodeCnt) when submitting jobs
by WANG, Hongying 03 May '24

03 May '24

Hi all, I have a question about the reservation. If I create a reservation with 3 nodes (NodeCnt=3), how can I use more than 3 nodes when submitting a job with --reservation=$RES_NAME ? Thanks a lot. Regards, -- WANG, Hongying ClusterTech Limited Advanced I.T. Solutions and Consultancy SINCE 2000 Cloud + High Performance Computing + AI & Analytics + Big Data www.clustertech.com Units 211-213, Lakeside 1, No. 8 Science Park West Avenue, Hong Kong Science Park, N.T. Hong Kong Hong Kong • Beijing • Shanghai • Guangzhou • Shenzhen • Xi'an • Sydney

1 0

scheduling according time requirements
by Dietmar Rieder 30 Apr '24

30 Apr '24

Hi, is it possible to have slurm scheduling jobs automatical according to the "-t" time requirements to a fitting partition? e.g. 3 partitions PartitionName=standard Nodes=c-[01-10] Default=YES MaxTime=04:00:00 DefaultTime=00:10:00 State=UP OverSubscribe=NO PartitionName=medium Nodes=c-[04-08] Default=NO MaxTime=24:00:00 DefaultTime=04:00:00 State=UP OverSubscribe=NO PartitionName=long Nodes=c-[09-10] Default=NO MaxTime=336:00:00 DefaultTime=24:00:00 State=UP OverSubscribe=NO So in the standard partition which is the default we have all nodes and a max time of 4h, in the medium partition we have 4 nodes with a max time of 24h and in the long partition we have 2 nodes with a max time of 336h. I was hoping that if I submit a job with -t 01:00:00 it can be run on any node (standard partition), whereas when specifying -t 05:00:00 or -t 48:00:00 the job will run on the nodes of the medium or long partition respectively. However, my job will not get scheduled at all when -t is greater than 01:00:00 i.e. ]$ srun --cpus-per-task 1 -t 01:00:01 --pty bash srun: Requested partition configuration not available now srun: job 42095 queued and waiting for resources it will wait forever because the standard partition is selected, I was thinking that slurm would automatically switch to the medium partition. Do I misunderstand something there? Or can this be somehow configured. Thanks so much and sorry for the naive question Dietmar

4 10

Trying to Track Down root Usage
by Jason Simms 29 Apr '24

29 Apr '24

Hello all, Each week, I generate an automated report of the top users by CPU hours. This week, for whatever reason the user root accounted for a massive number of hours: -------------------------------------------------------- Login Proper Name Used Account --------------- -------------------- --------- --------- root root 28224 root I've tried to generate a report on what might account for this, but so far I'm stumped. For example, when I run a typical report: sacct -o jobid,jobname,start,end,NNodes,NCPUS,ReqMem,CPUTime,AveRSS,MaxRSS -S 2024-04-22 -E 2024-04-29 --user=root --units=G Nothing comes up (same if I change --user=root to --account=root). I'd appreciate any thoughts as to how to investigate or understand this. It may well be that I'm missing something fundamental. Warmest regards, Jason -- *Jason L. Simms, Ph.D., M.P.H.* Manager of Research Computing Swarthmore College Information Technology Services (610) 328-8102 Schedule a meeting: https://calendly.com/jlsimms

2 2

HPC Principal System Engineer at the Broad
by Paul Edmon 25 Apr '24

25 Apr '24

A friend ask me to pass this along. Figured some folks on this list might be interested. https://broadinstitute.avature.net/en_US/careers/JobDetail/HPC-Principal-Sy… -Paul Edmon-

1 0

Apply an specific QoS to all users that belongs to an specific account
by Gestió Servidors 23 Apr '24

23 Apr '24

Hi, I would like to know if it is possible to apply an specific QoS to all users that belongs to an specific account. For example, I have created some new users "user_XX" and, also, I have created their new accounts in SLURM with "sacctmgr create account name=Test" and "sacctmgr create user name=user_XX DefaultAccount=Test". After it, I have changed default QoS "normal" to a new QoS "minimal" (with some limits) to account "Test" (sacctmgr modify account where name=Test set qos=minimal), but what I have seen is that users "user_XX" that belongs to "Test" account continue in QoS "normal" (QoS by default), so it seems that users have not inherited QoS applied to their account. Is there any way to do this and am I doing something wrong? Thanks. -- [cid:image003.jpg@01DA9569.AE233AA0]<http://www.uab.cat/> Daniel Ruiz Molina Tècnic Mitjà Informàtic Arquitectura de Computadors i Sistemes Operatius Escola d'Enginyeria Edifici Q - Despatx QC/3052 - Carrer de les Sitges Campus de la UAB · 08193 Bellaterra (Cerdanyola del Vallès) · Barcelona · Spain +34 93 581 35 44 www.uab.cat<http://www.uab.cat/> Daniel Ruiz at UAB<https://tinyurl.com/yd95zb8j> [cid:image004.jpg@01DA9569.AE233AA0]<www.linkedin.com/in/daniel-ruiz-molina-50a83b27> Aquest missatge s'adreça exclusivament a la persona destinatària i pot contenir informació privada o confidencial. Si l'heu rebut per error, comuniqueu-nos-ho i destruïu-lo, i tingueu present que no teniu autorització per fer-ne cap ús. Abans d'imprimir aquest missatge penseu en el medi ambient.

1 0

Integrating Slurm with WekaIO
by Jeffrey Layton 19 Apr '24

19 Apr '24

Good afternoon, I'm working on a cluster of NVIDIA DGX A100's that is using BCM 10 (Base Command Manager which is based on Bright Cluster Manager). I ran into an error and only just learned that Slurm and Weka don't get along (presumably because Weka pins their client threads to cores). I read through their documentation here: https://docs.weka.io/best-practice-guides/weka-and-slurm-integration#headin… I through I set everything correctly but when I try to restart the slurm server I get the following: Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: slurmd: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: slurmd: error: fetch_config: DNS SRV lookup failed Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: slurmd: error: _establish_configuration: failed to load configs Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: slurmd: error: slurmd initialization failed Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: error: fetch_config: DNS SRV lookup failed Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: error: _establish_configuration: failed to load configs Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: error: slurmd initialization failed Apr 19 05:29:39 bcm10-headnode systemd[1]: slurmd.service: Main process exited, code=exited, status=1/FAILURE Apr 19 05:29:39 bcm10-headnode systemd[1]: slurmd.service: Failed with result 'exit-code'. Has anyone encountered this? I read this is usually associated with configless Slurm, but I don't know how Slurm is built in BCM. slurm.conf is located in /cm/shared/apps/slurm/var/etc/slurm and this is what I edited. The environment variables for Slurm are set correctly so it points to this slurm.conf file. One thing that I did not do was tell Slurm which cores Weka was using. I can seem to figure out the syntax for this. Can someone share the changes they made to slurm.conf? Thanks! Jeff

3 5

Munge log-file fills up the file system to 100%
by Ole Holm Nielsen 19 Apr '24

19 Apr '24

We have some new AMD EPYC compute nodes with 96 cores/node running RockyLinux 8.9. We've had a number of incidents where the Munge log-file /var/log/munge/munged.log suddenly fills up the root file system, after a while to 100% (tens of GBs), and the node eventually comes to a grinding halt! Wiping munged.log and restarting the node works around the issue. I've tried to track down the symptoms and this is what I found: 1. In munged.log there are infinitely many lines filling up the disk: 2024-04-11 09:59:29 +0200 Info: Suspended new connections while processing backlog 2. The slurmd is not getting any responses from munged, even though we run "munged --num-threads 10". The slurmd.log displays errors like: [2024-04-12T02:05:45.001] error: If munged is up, restart with --num-threads=10 [2024-04-12T02:05:45.001] error: Munge encode failed: Failed to connect to "/var/run/munge/munge.socket.2": Resource temporarily unavailable [2024-04-12T02:05:45.001] error: slurm_buffers_pack_msg: auth_g_create: RESPONSE_ACCT_GATHER_UPDATE has authentication error 3. The /var/log/messages displays the errors from slurmd as well as NetworkManager saying "Too many open files in system". The telltale syslog entry seems to be: Apr 12 02:05:48 e009 kernel: VFS: file-max limit 65536 reached where the limit is confirmed in /proc/sys/fs/file-max. We have never before seen any such errors from Munge. The error may perhaps be triggered by certain user codes (possibly star-ccm+) that might be opening a lot more files on the 96-core nodes than on nodes with a lower core count. My workaround has been to edit the line in /etc/sysctl.conf: fs.file-max = 131072 and update settings by "sysctl -p". We haven't seen any of the Munge errors since! The version of Munge in RockyLinux 8.9 is 0.5.13, but there is a newer version in https://github.com/dun/munge/releases/tag/munge-0.5.16 I can't figure out if 0.5.16 has a fix for the issue seen here? Questions: Have other sites seen the present Munge issue as well? Are there any good recommendations for setting the fs.file-max parameter on Slurm compute nodes? Thanks for sharing your insights, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

4 10

Reserving resources for use by non-slurm stuff
by Shooktija S N 17 Apr '24

17 Apr '24

Hi, I am running Slurm (v22.05.8) on 3 nodes each with the following specs: OS: Proxmox VE 8.1.4 x86_64 (based on Debian 12) CPU: AMD EPYC 7662 (128) GPU: NVIDIA GeForce RTX 4070 Ti Memory: 128 Gb This is /etc/slurm/slurm.conf on all 3 computers without the comment lines: ClusterName=DlabCluster SlurmctldHost=server1 GresTypes=gpu ProctrackType=proctrack/linuxproc ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=root StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/affinity,task/cgroup InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 SchedulerType=sched/backfill SelectType=select/cons_tres JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=debug3 SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=debug3 SlurmdLogFile=/var/log/slurmd.log NodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1 PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP I want to reserve a few cores and a few gigs of RAM for use only by the OS which cannot be accessed by jobs being managed by Slurm. What configuration do I need to do to achieve this? Is it possible to reserve in a similar fashion a 'percent' of the GPU which Slurm cannot exceed so that the OS has some GPU resources? Is it possible to have these configs be different for each of the 3 nodes? Thanks!

3 2

Inconsistencies in CPU time Reporting by sreport and sacct Tools
by KK 17 Apr '24

17 Apr '24

I wish to ascertain the CPU core time utilized by user dj1 and dj. I have tested with sreport cluster UserUtilizationByAccount, sreport job SizesByAccount, and sacct. It appears that sreport cluster UserUtilizationByAccount displays the total core hours used by the entire account, rather than the individual user's cpu time. Here are the specifics: Users dj and dj1 are both under the account mehpc. In 2024-04-12 ~ 2024-04-15, dj1 used approximately 10 minutes of core time, while dj used about 4 minutes. However, "sreport Cluster UserUtilizationByAccount user=dj1 start=2024-04-12 end=2024-04-15" shows 14 minutes of usage. Similarly, "sreport job SizesByAccount Users=dj start=2024-04-12 end=2024-04-15" hows about 14 minutes. Using "sreport job SizesByAccount Users=dj1 start=2024-04-12 end=2024-04-15" or "sacct -u dj1 -S 2024-04-12 -E 2024-04-15 -o "jobid,partition,account,user,alloccpus,cputimeraw,state,workdir%60" -X |awk 'BEGIN{total=0}{total+=$6}END{print total}'" yields the accurate values, which are around 10 minutes for dj1. Attachment are the details.

1 0

2025

2024