- slurm-users - lists.schedmd.com

Problems with gres.conf
by Gestió Servidors 10 May '24

10 May '24

Hello, I am trying to rewrite my gres.conf file. Before changes, this file was just like this: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 # you can seee that nodes node-gpu-1 and node-gpu-2 have two GPUs each one, whereas nodes node-gpu-3 and node-gpu-4 have only one GPU each one And my slurmd.conf was this: [...] AccountingStorageTRES=gres/gpu GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 [...] With this configuration, all seems works fine, except slurmctld.log reports: [...] error: _node_config_validate: gres/gpu: invalid GRES core specification (0-11) on node node-gpu-3 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-1 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-2 error: _node_config_validate: gres/gpu: invalid GRES core specification (0-7) on node node-gpu-4 [...] However, even these errors, users can submit jobs and request GPUs resources. Now, I have tried to reconfigure gres.conf and slurmd.conf in this way: gres.conf: Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 # there is no NodeName attribute slurmd.conf: [...] NodeName=node-gpu-1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 # there is no CPUs attribute [...] With this new configuration, nodes with GPU start correctly slurmd.service daemon, but nodes without GPU (node-worker-[0-22]) can't start slurmd.service daemon and returns this error: [...] error: Waiting for gres.conf file /dev/nvidia0 fatal: can't stat gres.conf file /dev/nvidia0: No such file or directory [...] It seems SLURM is waiting that "node-workers" have also an nvidia GPU but not, theses nodes haven't GPU... So, where is my configuration error? I have read in https://slurm.schedmd.com/gres.conf.html about syntax and examples but it seems I'm doing some wrong. Thanks!!

1 0

Re: srun launched mpi job occasionally core dumps
by Henderson, Brent 09 May '24

09 May '24

Thanks for the suggestion Ole - I'll see if I can get that in the mix to try over the next few days. I can report that 23.02.7 tree had the same issues, so going backwards on the slurm bits did not have any impact. Brent

1 0

Partition Preemption Configuration Question
by Jason Simms 09 May '24

09 May '24

Hello all, The Slurm docs have me a bit confused... I'm wanting to enable job preemption on certain partitions but not others. I *presume* I would set PreemptType=preempt/partition_prio globally, but then on the partitions where I don't want jobs to be able to be preempted, I would set PreemptMode=off within the configuration for that specific partition. The documentation, however, says that setting PreemptMode=off at a partition level "is only compatible with PreemptType=preempt/none at a global level" yet then immediately says that doing so is a "common use case for this parameter is to set it on a partition to disable preemption for that partition," which indicates preemption would still be allowable for other partitions. If PreemptType is set to preempt/none globally, and I *cannot* set that as an option for a given partition (at least, the documentation doesn't indicate that is a valid parameter for a partition), wouldn't preemption be disabled globally anyway? The wording seems odd to me and almost contradictory. Is it possible to have PreemptType=preempt/partition_prio set globally, yet also disable it on specific partitions with PreemptMode=off? Is PreemptType actually a valid configuration option for specific partitions? Thanks for any guidance. Warmest regards, Jason -- *Jason L. Simms, Ph.D., M.P.H.* Manager of Research Computing Swarthmore College Information Technology Services (610) 328-8102 Schedule a meeting: https://calendly.com/jlsimms

3 3

Container Jobs "hanging"
by sean+slurm＠superorbital.io 09 May '24

09 May '24

Hello. I am new to this list and Slurm overall. I have a lot of experience in computer operations, including Kubernetes, but I am currently exploring Slurm in some depth. I have set up a small cluster and, in general, have gotten things working, but when I try to run a container job, it runs the command but then appears to hang as if the job container is still running. So, running the following works, but it never returns to the prompt unless I use [Control-C]. $ srun --container /shared_fs/shared/oci_images/alpine uptime 19:21:47 up 20:43, 0 users, load average: 0.03, 0.25, 0.15 I'm unsure if something is misconfigured or if I'm misunderstanding how this should work, but any help and/or pointers would be greatly appreciated. Thanks! Sean

1 0

scrontab question
by Sandor 09 May '24

09 May '24

I am working out the details of scrontab. My initial testing is giving me an unsolvable question Within scrontab editor I have the following example from the slurm documentation: 0,5,10,15,20,25,30,35,40,45,50,55 * * * * /directory/subdirectory/crontest.sh When I save it, scrontab marks the line with #BAD: I do not understand why. The only difference I have is the directory structure. Is there an underlying assumption that traditional Linux crontab is available to the general user?

4 3

Slurm With Podman - No child processes error
by ARNULD 08 May '24

08 May '24

I have integrated Podman with Slurm as per the docs ( https://slurm.schedmd.com/containers.html#podman-scrun) and when I do a test run: "podman run hello-world" (this runs fine) ---- $ podman run alpine hostname executable file `/usr/bin/hostname` not found in $PATH: No such file or directory srun: error: slurm1: task 0: Exited with exit code 1 ----- $ podman run alpine printenv SLURM_JOB_ID executable file `/usr/bin/printenv` not found in $PATH: No such file or directory srun: error: slurm1: task 0: Exited with exit code 1 scrun: error: run_command_waitpid_timeout: waitpid(67537): No child processes --- podman run alpine uptime 11:31:28 up 5:32, 0 users, load average: 0.00, 0.00, 0.00 scrun: error: run_command_waitpid_timeout: waitpid(68160): No child processes ------ I built a small image from python:alpine3.19 which just prints "hello world" and numbers from 1 to 10. Here is a run: $ podman run -it --rm hello-python $ podman run -it --rm hello-python Hello, world! Numbers from 1 to 10: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] ---- No error with my image. Also I tested podman on another machine without Slurm. Podman with its default runtime prints the hostname fine with "podman run alpine hostname". So something to do with its integration with Slurm. What can I do to diagnose the problem?

1 0

StateSaveLocation and Slurm HA
by Pierre Abele 07 May '24

07 May '24

Hi all, I am looking for a clean way to set up Slurms native high availability feature. I am managing a Slurm cluster with one control node (hosting both slurmctld and slurmdbd), one login node and a few dozen compute nodes. I have a virtual machine that I want to set up as a backup control node. The Slurm documentation says the following about the StateSaveLocation directory: > The directory used should be on a low-latency local disk to prevent file system delays from affecting Slurm performance. If using a backup host, the StateSaveLocation should reside on a file system shared by the two hosts. We do not recommend using NFS to make the directory accessible to both hosts, but do recommend a shared mount that is accessible to the two controllers and allows low-latency reads and writes to the disk. If a controller comes up without access to the state information, queued and running jobs will be cancelled. [1] My question: How do I implement the shared file system for the StateSaveLocation? I do not want to introduce a single point of failure by having a single node that hosts the StateSaveLocation, neither do I want to put that directory on the clusters NFS storage since outages/downtime of the storage system will happen at some point and I do not want that to cause an outage of the Slurm controller. Any help or ideas would be appreciated. Best, Pierre [1] https://slurm.schedmd.com/quickstart_admin.html#Config -- Pierre Abele, M.Sc. HPC Administrator Max-Planck-Institute for Evolutionary Anthropology Department of Primate Behavior and Evolution Deutscher Platz 6 04103 Leipzig Room: U2.80 E-Mail: pierre_abele(a)eva.mpg.de Phone: +49 (0) 341 3550 245

3 2

"token expired" errors with auth/slurm
by Fabio Ranalli 07 May '24

07 May '24

Hi there, We've updated to 23.11.6 and replaced MUNGE with SACK. Performance and stability have both been pretty good, but we're occasionally seeing this in the slurmctld.log /[2024-05-07T03:50:16.638] error: decode_jwt: token expired at 1715053769 [2024-05-07T03:50:16.638] error: cred_p_unpack: decode_jwt() failed [2024-05-07T03:50:16.638] error: Malformed RPC of type REQUEST_BATCH_JOB_LAUNCH(4005) received [2024-05-07T03:50:16.641] error: slurm_receive_msg_and_forward: [[headnode.internal]:58286] failed: Header lengths are longer than data received [2024-05-07T03:50:16.648] error: service_connection: slurm_receive_msg: Header lengths are longer than data received/ it seems to impact a subset of nodes: jobs get killed and no new ones are allocated. Full functionality can be restored by simply restarting slurmctld first, and then slurmd. Is the token expected to actually expire? I didn't see this possibility mentioned in the docs. The problem occurs on an R&D cloud cluster based on EL9, with a pretty "flat" setup. _headnode_: configless slurmctld, slurmdbd, mariadb, nfsd _elastic compute nodes_: autofs, slurmd *//etc/slurm/slurm.conf/* AuthType=auth/slurm AuthInfo=use_client_ids CredType=cred/slurm *//etc/slurm/slurmdbd.conf/* AuthType=auth/slurm AuthInfo=use_client_ids Has anyone else encountered the same error? Thanks, Fabio -- *Fabio Ranalli* | Principal Systems Administrator Schrödinger, Inc. <https://schrodinger.com>

1 0

FreeBSD/aarch64: ld: error: unknown emulation: elf_aarch64
by Nuno Teixeira 07 May '24

07 May '24

Hello, I'm new maintainer of slurm FreeBSD port <https://www.freshports.org/sysutils/slurm-wlm/> and for a long time it failing on aarch64 platform. Any clues? > ld: error: unknown emulation: elf_aarch64 `clang -print-targets`: Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V thumb - Thumb thumbeb - Thumb (big endian) x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 Full log: https://pkg-status.freebsd.org/ampere2/data/main-arm64-default/pb78025a96ed… Thanks, -- Nuno Teixeira FreeBSD UNIX: <eduardo(a)FreeBSD.org> Web: https://FreeBSD.org

3 7

Rootless Docker Errors with Slurm
by ARNULD 06 May '24

06 May '24

I am trying to integrate Rootless Docker with Slurm. have set-up Rootless Docker as per the docs "https://slurm.schedmd.com/containers.html" . I have scrum.lua, oci.conf (for crun) and slurm.conf in place. Then "~/.config/docker/daemon.json" and "~/.config/systemd/user/docker.service.d/override.conf" are in place too. But I can't seem to get it to work: $ docker run $DOCKER_SECURITY alpine /bin/printenv SLURM_JOB_ID Unable to find image 'alpine:latest' locally latest: Pulling from library/alpine 4abcf2066143: Pull complete Digest: sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b Status: Downloaded newer image for alpine:latest docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/user/1000/docker-exec/containerd/daemon/io.containerd.runtime.v2.task/moby/97e8dd767977ac03ab7af54c015c0fd5dfd26e737771b977acb7e41f799023aa/log.json: no such file or directory): /usr/bin/scrun did not terminate successfully: exit status 1: unknown. One thing is if I don't use Slurm as the runtime for Docker (if I remove the " ~/.config/docker/daemon.json") then docker runs fine

1 0

2025

2024