- slurm-users - lists.schedmd.com

need to set From: address for slurm
by Vanhorn, Mike 07 Jun '24

07 Jun '24

All, When the slurm daemon is sending out emails, they are coming from “slurm(a)servername.subdomain.domain.edu<mailto:slurm@servername.subdomain.domain.edu>”. This has worked okay in the past, but due to a recent mail server change (over which I have no control whatsoever) this will no longer work. Now, the From: address is going to have to be something like “slurm-servername(a)domain.xn--edu-9o0a , or, at least something that ends in “(a)domain.xn--edu-9o0a (the subdomain being present will cause it to get rejected by the mail server. I am not seeing in the documentation how to change the “From:” address tha slurm uses. Is there a way to do this and I’m just missing it? --- Mike VanHorn Senior Computer Systems Administrator College of Engineering and Computer Science Wright State University 265 Russ Engineering Center 937-775-5157 michael.vanhorn(a)wright.edu

2 2

Re: Not being able to ssh to node with running job
by Ratnasamy, Fritz 06 Jun '24

06 Jun '24

So the squeue issue was resolved and was due to the partition being hidden. Unhiding it solves the problem. However, the ssh issue remains (looks like both were separate issues). The pam_slurm_adopt is working on all the other nodes but not on the new ones. Any idea how to solve this? Best, *Fritz Ratnasamy* Data Scientist Information Technology On Thu, Jun 6, 2024 at 2:11 PM Ratnasamy, Fritz via slurm-users < slurm-users(a)lists.schedmd.com> wrote: > As admin on the cluster, we do not observe any issue on our newly added > gpu nodes. > However, for regular users, they are not seeing their jobs running on > these gpu nodes when running squeue -u <username> ( it is > however showing as running status with sacct) and they are not able to ssh > to these newly added nodes when they have a running job on it. > I am not sure if these 2 are related (not being to ssh to mgpu node with > a running job on it and not listing a job with squeue for a user on the > same node). There are no issues reported on the other nodes. Anyone know > what is happening? > Best, > > *Fritz Ratnasamy* > > Data Scientist > > Information Technology > > > CAUTION: This email has originated outside of University email systems. > Please do not click links or open attachments unless you recognize the > sender and trust the contents as safe. > >

1 0

Re: JWKS File Update - Reread File
by Pedro Daniel Da Rocha Santos 06 Jun '24

06 Jun '24

Thanks for the reply. I was not aware of that change in 23.11. I am still using Slurm 23.02.7. Best regards, Pedro D.R. Santos Specialist, Ph.D., High Performance Computing Vestas Technology Centre Porto Vestas Wind Systems A/S Vestas will as part of your communication, interaction and business relationship with us collect and process personal data about you. You can read more about Vestas' collection and processing of your personal data and your rights as a data subject in our Privacy Policy<https://www.vestas.com/en/about/profile/privacy-policy>. Classification: Restricted

1 0

Not being able to ssh to node with running job
by Ratnasamy, Fritz 06 Jun '24

06 Jun '24

As admin on the cluster, we do not observe any issue on our newly added gpu nodes. However, for regular users, they are not seeing their jobs running on these gpu nodes when running squeue -u <username> ( it is however showing as running status with sacct) and they are not able to ssh to these newly added nodes when they have a running job on it. I am not sure if these 2 are related (not being to ssh to mgpu node with a running job on it and not listing a job with squeue for a user on the same node). There are no issues reported on the other nodes. Anyone know what is happening? Best, *Fritz Ratnasamy* Data Scientist Information Technology

1 0

JWKS File Update - Reread File
by Pedro Daniel Da Rocha Santos 06 Jun '24

06 Jun '24

Hello everyone, I am currently implementing a service to monitor for jwks key rotation. If a change is detected, the jwks file is updated. Is it enough to use "scontrol reconfigure" to reread the jwks file? Or does it require a full restart of the slurmctl daemon? Thanks in advance for the inputs. Best regards, Pedro D.R. Santos Specialist, Ph.D., High Performance Computing Vestas Technology Centre Porto Vestas Wind Systems A/S +351 924 241 907 pddro(a)vestas.com<mailto:pddro@vestas.com> http://www.vestas.com<http://www.vestas.com/> Company reg. name: Vestas Wind Systems A/S This e-mail is subject to our e-mail disclaimer statement. Please refer to www.vestas.com/legal/notice<http://www.vestas.com/legal/notice> If you have received this e-mail in error please contact the sender. Vestas will as part of your communication, interaction and business relationship with us collect and process personal data about you. You can read more about Vestas' collection and processing of your personal data and your rights as a data subject in our Privacy Policy<https://www.vestas.com/en/about/profile/privacy-policy>. Classification: Restricted

2 1

diagnosing why interactive/non-interactive job waits are so long with State=MIXED
by Robert Kudyba 05 Jun '24

05 Jun '24

At the moment we have 2 nodes that are having long wait times. Generally this is when the nodes are fully allocated. What would be the other reasons if there is still enough available memory and CPU available, that a job would take so long? Slurm version is 23.02.4 via Bright Computing. Note the compute nodes have hyperthreading enabled but that should be irrelevant. Is there a way to determine what else could be holding jobs up? srun --pty -t 0-01:00:00 --nodelist=node001 --gres=gpu:1 -A ourts -p short /bin/bash srun: job 672204 queued and waiting for resources scontrol show node node001 NodeName=m001 Arch=x86_64 CoresPerSocket=48 CPUAlloc=24 CPUEfctv=192 CPUTot=192 CPULoad=20.37 AvailableFeatures=location=local ActiveFeatures=location=local Gres=gpu:A6000:8 NodeAddr=node001 NodeHostName=node001 Version=23.02.4 OS=Linux 5.14.0-70.13.1.el9_0.x86_64 #1 SMP PREEMPT Thu Apr 14 12:42:38 EDT 2022 RealMemory=1031883 AllocMem=1028096 FreeMem=222528 Sockets=2 Boards=1 State=MIXED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=ours,short BootTime=2024-04-29T16:18:30 SlurmdStartTime=2024-05-18T16:48:11 LastBusyTime=2024-06-03T10:49:49 ResumeAfterTime=None CfgTRES=cpu=192,mem=1031883M,billing=192,gres/gpu=8 AllocTRES=cpu=24,mem=1004G,gres/gpu=2,gres/gpu:a6000=2 CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s grep 672204 /var/log/slurmctld [2024-06-04T15:50:35.627] sched: _slurm_rpc_allocate_resources JobId=672204 NodeList=(null) usec=852

3 5

Re: Problems with gres.conf
by Gestió Servidors 05 Jun '24

05 Jun '24

Hi, my GPU testing system (named “gpu-node”) is a simple computer with one socket and a processor " Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz". Executing "lscpu", I can see there are 4 cores per socket, 2 threads per core and 8 CPUs: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 26 Model name: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz My “gres.conf” file is: NodeName=gpu-node Name=gpu Type=GeForce-GTX-TITAN-X File=/dev/nvidia0 CPUs=0-1 NodeName=gpu-node Name=gpu Type=GeForce-GTX-TITAN-Black File=/dev/nvidia1 CPUs=2-3 Running “numactl -H” in “gpu-node” host, reports: available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 node 0 size: 7809 MB node 0 free: 6597 MB node distances: node 0 0: 10 CPUs are assigned 0-1 for first GPU and 2-3 for second GPU. However, “lscpu” shows 8 CPUs… If I rewrite “gres.conf” in this way: NodeName=gpu-node Name=gpu Type=GeForce-GTX-TITAN-X File=/dev/nvidia0 CPUs=0-3 NodeName=gpu-node Name=gpu Type=GeForce-GTX-TITAN-Black File=/dev/nvidia1 CPUs=4-7 when I run “scontrol reconfigure”, slurmctld log reports this error message: [2024-06-05T11:42:18.558] error: _node_config_validate: gres/gpu: invalid GRES core specification (4-7) on node gpu-node So I think SLURM only can get physical cores and not threads, so my node only can serve 4 cores (in “lspcu”) but in gres.conf I need to write “CPUs”, not “Cores”… isn’t it? But if “numactl -H” shows 8 CPUs, why I can use this 8 CPUs in “gres.conf”? Sorry about this large email. Thanks.

1 0

Problems with gres.conf
by Gestió Servidors 04 Jun '24

04 Jun '24

Hello, I am trying to rewrite my gres.conf file. Before changes, this file was just like this: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 # you can seee that nodes node-gpu-1 and node-gpu-2 have two GPUs each one, whereas nodes node-gpu-3 and node-gpu-4 have only one GPU each one And my slurmd.conf was this: [...] AccountingStorageTRES=gres/gpu GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 [...] With this configuration, all seems works fine, except slurmctld.log reports: [...] error: _node_config_validate: gres/gpu: invalid GRES core specification (0-11) on node node-gpu-3 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-1 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-2 error: _node_config_validate: gres/gpu: invalid GRES core specification (0-7) on node node-gpu-4 [...] However, even these errors, users can submit jobs and request GPUs resources. Now, I have tried to reconfigure gres.conf and slurmd.conf in this way: gres.conf: Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 # there is no NodeName attribute slurmd.conf: [...] NodeName=node-gpu-1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 # there is no CPUs attribute [...] With this new configuration, nodes with GPU start correctly slurmd.service daemon, but nodes without GPU (node-worker-[0-22]) can't start slurmd.service daemon and returns this error: [...] error: Waiting for gres.conf file /dev/nvidia0 fatal: can't stat gres.conf file /dev/nvidia0: No such file or directory [...] It seems SLURM is waiting that "node-workers" have also an nvidia GPU but not, theses nodes haven't GPU... So, where is my configuration error? I have read in https://slurm.schedmd.com/gres.conf.html about syntax and examples but it seems I'm doing some wrong. Thanks!!

2 1

Tracking GPU utilization with sacct
by Robertson, Jordan 03 Jun '24

03 Jun '24

Hi all, I’m trying to pull (and understand) some GPU usage metrics for historical purposes, and dug into sacct’s TRES reporting a bit. We have AccountingStorageTRES=gres/gpu set in slurm.conf so we do see gres/gpuutil and gres/gpumem numbers available, but I’m struggling to find Slurm-side documentation that describes the units of these values. In looking at the code for gpu_nvml.c it seems the “nvmlDeviceGetProcessUtilization” function is being used and returns units in percentages, but I’m lost on the rest of the calculation. Does anyone know if these units are percentages, and how they are calculated for the final job record, especially wrt multi-GPU jobs with a bunch of processes/moving parts? For context I’ve been looking at TRESUsageInTot and TRESUsageInAve so far. Also we’re currently running Slurm v23.02.6 Thanks in advance! -- Jordan Robertson Preferred pronouns: he/him/his Technology Architect | Research Technology Services DigITs, Technology Division Memorial Sloan Kettering Cancer Center 929-687-1066 robertj8(a)mskcc.org<mailto:robertj8@mskcc.org> ===================================================================== Please note that this e-mail and any files transmitted from Memorial Sloan Kettering Cancer Center may be privileged, confidential, and protected from disclosure under applicable law. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any reading, dissemination, distribution, copying, or other use of this communication or any of its attachments is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting this message, any attachments, and all copies and backups from your computer. Disclaimer ID:MSKCC

1 0

sbatch and --nodes
by Michael DiDomenico 31 May '24

31 May '24

its friday and i'm either doing something silly or have a misconfig somewhere, i can't figure out which when i run sbatch --nodes=1 --cpus-per-task=1 --array=1-100 --output test_%A_%a.txt --wrap 'uname -n' sbatch doesn't seem to be adhering to the --nodes param. when i look at my output files it's spreading them across more nodes. in the simple case above it's 50/50, but if i through a random sleep in, it'll be more. and if i expand the array it'll use even more nodes. i'm using con/tres and have cr_core_memory,cr_one_core_per_task set

2 1

Re: Container Jobs "hanging"
by Joshua Randall 31 May '24

31 May '24

Hi Sean, I appear to be having the same issue that you are having with OCI container jobs running forever / appearing to hang. I haven't figured it out yet, but perhaps we can compare notes and determine what aspect of configuration we both share. Like you, I was following the examples in https://slurm.schedmd.com/containers.html and originally encountered the issue with an alpine container image running the `uptime` command, but I have also confirmed the issue with other images including ubuntu and with other processes. I always get the same results - the container process runs to completion and exits, but then the slurm job continues to run until it is cancelled or killed. I have slurm v23.11.6 and am using the nvidia-container-runtime, what slurm version and runtime are you using? My oci.conf is: ``` $ cat /etc/slurm/oci.conf EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" RunTimeQuery="nvidia-container-runtime --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t" RunTimeKill="nvidia-container-runtime --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t" RunTimeDelete="nvidia-container-runtime --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t" RunTimeRun="nvidia-container-runtime --rootless=true --root=/run/user/%U/ run %n.%u.%j.%s.%t -b %b" ``` Hope that we can get to the bottom of this and resolve our issues with OCI containers! Josh. --- Hello. I am new to this list and Slurm overall. I have a lot of experience in computer operations, including Kubernetes, but I am currently exploring Slurm in some depth. I have set up a small cluster and, in general, have gotten things working, but when I try to run a container job, it runs the command but then appears to hang as if the job container is still running. So, running the following works, but it never returns to the prompt unless I use [Control-C]. $ srun --container /shared_fs/shared/oci_images/alpine uptime 19:21:47 up 20:43, 0 users, load average: 0.03, 0.25, 0.15 I'm unsure if something is misconfigured or if I'm misunderstanding how this should work, but any help and/or pointers would be greatly appreciated. Thanks! Sean -- slurm-users mailing list -- slurm...(a)lists.schedmd.com To unsubscribe send an email to slurm-us...(a)lists.schedmd.com -- Dr. Joshua C. Randall Principal Software Engineer Altos Labs email: jrandall(a)altoslabs.com -- Altos Labs UK Limited | England | Company reg 13484917 Registered address: 3rd Floor 1 Ashley Road, Altrincham, Cheshire, United Kingdom, WA14 2DT

1 1

slurmdbd not connecting to mysql (mariadb)
by Radhouane Aniba 31 May '24

31 May '24

Hi everyone I am trying to get slurmdbd to run on my local home server but I am really struggling. Note : am a novice slurm user my slurmdbd always times out even though all the details in the conf file are correct My log looks like this [2024-05-29T20:51:30.088] Accounting storage MYSQL plugin loaded [2024-05-29T20:51:30.088] debug2: ArchiveDir = /tmp [2024-05-29T20:51:30.088] debug2: ArchiveScript = (null) [2024-05-29T20:51:30.088] debug2: AuthAltTypes = (null) [2024-05-29T20:51:30.088] debug2: AuthInfo = (null) [2024-05-29T20:51:30.088] debug2: AuthType = auth/munge [2024-05-29T20:51:30.088] debug2: CommitDelay = 0 [2024-05-29T20:51:30.088] debug2: DbdAddr = localhost [2024-05-29T20:51:30.088] debug2: DbdBackupHost = (null) [2024-05-29T20:51:30.088] debug2: DbdHost = head-node [2024-05-29T20:51:30.088] debug2: DbdPort = 7032 [2024-05-29T20:51:30.088] debug2: DebugFlags = (null) [2024-05-29T20:51:30.088] debug2: DebugLevel = 6 [2024-05-29T20:51:30.088] debug2: DebugLevelSyslog = 10 [2024-05-29T20:51:30.088] debug2: DefaultQOS = (null) [2024-05-29T20:51:30.088] debug2: LogFile = /var/log/slurmdbd.log [2024-05-29T20:51:30.088] debug2: MessageTimeout = 100 [2024-05-29T20:51:30.088] debug2: Parameters = (null) [2024-05-29T20:51:30.088] debug2: PidFile = /run/slurmdbd.pid [2024-05-29T20:51:30.088] debug2: PluginDir = /usr/lib/x86_64-linux-gnu/slurm-wlm [2024-05-29T20:51:30.088] debug2: PrivateData = none [2024-05-29T20:51:30.088] debug2: PurgeEventAfter = 1 months* [2024-05-29T20:51:30.088] debug2: PurgeJobAfter = 12 months* [2024-05-29T20:51:30.088] debug2: PurgeResvAfter = 1 months* [2024-05-29T20:51:30.088] debug2: PurgeStepAfter = 1 months [2024-05-29T20:51:30.088] debug2: PurgeSuspendAfter = 1 months [2024-05-29T20:51:30.088] debug2: PurgeTXNAfter = 12 months [2024-05-29T20:51:30.088] debug2: PurgeUsageAfter = 24 months [2024-05-29T20:51:30.088] debug2: SlurmUser = root(0) [2024-05-29T20:51:30.089] debug2: StorageBackupHost = (null) [2024-05-29T20:51:30.089] debug2: StorageHost = localhost [2024-05-29T20:51:30.089] debug2: StorageLoc = slurm_acct_db [2024-05-29T20:51:30.089] debug2: StoragePort = 3306 [2024-05-29T20:51:30.089] debug2: StorageType = accounting_storage/mysql [2024-05-29T20:51:30.089] debug2: StorageUser = slurm [2024-05-29T20:51:30.089] debug2: TCPTimeout = 2 [2024-05-29T20:51:30.089] debug2: TrackWCKey = 0 [2024-05-29T20:51:30.089] debug2: TrackSlurmctldDown= 0 [2024-05-29T20:51:30.089] debug2: acct_storage_p_get_connection: request new connection 1 [2024-05-29T20:51:30.089] debug2: Attempting to connect to localhost:3306 [2024-05-29T20:51:30.090] slurmdbd version 19.05.5 started [2024-05-29T20:51:30.090] debug2: running rollup at Wed May 29 20:51:30 2024 [2024-05-29T20:51:30.091] debug2: Everything rolled up [2024-05-29T20:51:49.673] Terminate signal (SIGINT or SIGTERM) received [2024-05-29T20:51:49.673] debug: rpc_mgr shutting down my config file looks like this ArchiveEvents=yes ArchiveJobs=yes ArchiveResvs=yes ArchiveSteps=no ArchiveSuspend=no ArchiveTXN=no ArchiveUsage=no PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month PurgeTXNAfter=12month PurgeUsageAfter=24month # Authentication info AuthType=auth/munge # slurmDBD info DbdAddr=localhost DbdHost=head-node DbdPort=7032 SlurmUser=root MessageTimeout=100 DebugLevel=5 #DefaultQOS=normal,standby LogFile=/var/log/slurmdbd.log PidFile=/run/slurmdbd.pid #PrivateData=accounts,users,usage,jobs #TrackWCKey=yes # # Database info StorageType=accounting_storage/mysql StorageHost=localhost StoragePort=3306 StoragePass=slurmdbpass StorageUser=slurm StorageLoc=slurm_acct_db I used standard names and passwords to get started and I will change later but everytime I try to start slurmdbd.service it crashes and I have that log that I shared with you I use these versions slurmdbd -V slurm-wlm 19.05.5 mysql Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 Everything else Is working properly except I cannot get slurmdbd to work and at this point I exhausted all my possible trials :) looking for some expert insights :) Any idea what I am doing wrong here ? Also I didn't compile any slurm package. I used the binary from apt repos Any help will be appreciated Cheers Rad --

7 12

Re: slurmdbd not connecting to mysql (mariadb)
by Radhouane Aniba 31 May '24

31 May '24

manually running it through sudo slurmdbd -D /path/to/conf is very quick on my fresh install trying to start the slurmdbd through systemctl take 3 minutes and then crashes and fail Is there an alternative to systemctl to start the slurmdbd in the background ? But most importantly I wanted to know why it takes so long through systemctl. Maybe I can increase the timeout limit ? On Thu, May 30, 2024 at 11:54 PM Ryan Novosielski <novosirj(a)rutgers.edu> wrote: > It may take longer to start than systemd allows for. How long does it take > to start from the command line? It’s common to need to run it manually for > upgrades to complete. > > -- > #BlackLivesMatter > ____ > || \\UTGERS, |---------------------------*O*--------------------------- > ||_// the State | Ryan Novosielski - novosirj(a)rutgers.edu > || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus > || \\ of NJ | Office of Advanced Research Computing - MSB > A555B, Newark > `' > > On May 30, 2024, at 20:24, Radhouane Aniba via slurm-users < > slurm-users(a)lists.schedmd.com> wrote: > > Ok I made some progress here. > > I removed and purged slurmdbd mysql mariadb etc .. and started from > scratch. > I added the recommended mysqld requirements > > Started slurmdbd manually : sudo slurmdbd -D /path/to/conf and everything > worked well > > When I tried to start the service sudo systemctl start slurmdbd.service > it didnt work > > sudo systemctl status slurmdbd.service > ● slurmdbd.service - Slurm DBD accounting daemon > Loaded: loaded (/etc/systemd/system/slurmdbd.service; enabled; vendor > preset: enabled) > Active: failed (Result: timeout) since Fri 2024-05-31 00:21:30 UTC; > 2min 5s ago > Process: 6258 ExecStart=/usr/sbin/slurmdbd -D > /etc/slurm-llnl/slurmdbd.conf (code=exited, status=0/SUCCESS) > > May 31 00:20:00 hannibal-hn systemd[1]: Starting Slurm DBD accounting > daemon... > May 31 00:21:30 hannibal-hn systemd[1]: slurmdbd.service: start operation > timed out. Terminating. > May 31 00:21:30 hannibal-hn systemd[1]: slurmdbd.service: Failed with > result 'timeout'. > May 31 00:21:30 hannibal-hn systemd[1]: Failed to start Slurm DBD > accounting daemon. > > Even though it is the same command ?! > > Any idea ? > > > On Thu, May 30, 2024 at 5:02 PM Radhouane Aniba <aradwen(a)gmail.com> wrote: > >> Thank you Ahmet and Brian, >> >> Ahmet, which conf in particular slurmdbd is readiugn from, I parsed all >> the cnf files for mysql and I cannot find the data it is displaying here >> >> slurmdbd: debug2: Attempting to connect to localhost:3306 >> slurmdbd: debug2: innodb_buffer_pool_size: 134217728 >> slurmdbd: debug2: innodb_log_file_size: 50331648 >> slurmdbd: debug2: innodb_lock_wait_timeout: 50 >> slurmdbd: error: Database settings not recommended values: >> innodb_buffer_pool_size innodb_lock_wait_timeout >> >> >> sudo tree /etc/mysql/* >> /etc/mysql/conf.d >> ├── mysql.cnf >> └── mysqldump.cnf >> /etc/mysql/debian.cnf >> /etc/mysql/debian-start >> /etc/mysql/FROZEN >> /etc/mysql/mariadb.cnf >> /etc/mysql/mariadb.conf.d >> ├── 50-client.cnf >> ├── 50-mysql-clients.cnf >> ├── 50-mysqld_safe.cnf >> └── 50-server.cnf >> /etc/mysql/my.cnf >> /etc/mysql/my.cnf.fallback >> /etc/mysql/mysql.cnf >> /etc/mysql/mysql.conf.d >> ├── mysql.cnf >> └── mysqld.cnf >> >> On Thu, May 30, 2024 at 12:21 PM Brian Andrus via slurm-users < >> slurm-users(a)lists.schedmd.com> wrote: >> >>> That SIGTERM message means something is telling slurmdbd to quit. >>> >>> Check your cron jobs, maintenance scripts, etc. Slurmdbd is being told >>> to shutdown. If you are running in the foreground, a ^C does that. If you >>> run a kill or killall on it, you will get that same message. >>> >>> Brian Andrus >>> On 5/30/2024 6:53 AM, Radhouane Aniba via slurm-users wrote: >>> >>> Yes I can connect to my database using mysql --user=slurm >>> --password=slurmdbpass slurm_acct_db and there is no firewall blocking >>> mysql after checking the firewall question >>> >>> ALso here is the output of slurmdbd -D -vvv (note I can only run this as >>> sudo ) >>> >>> sudo slurmdbd -D -vvv >>> slurmdbd: debug: Log file re-opened >>> slurmdbd: debug: Munge authentication plugin loaded >>> slurmdbd: debug2: mysql_connect() called for db slurm_acct_db >>> slurmdbd: debug2: Attempting to connect to localhost:3306 >>> slurmdbd: debug2: innodb_buffer_pool_size: 134217728 >>> slurmdbd: debug2: innodb_log_file_size: 50331648 >>> slurmdbd: debug2: innodb_lock_wait_timeout: 50 >>> slurmdbd: error: Database settings not recommended values: >>> innodb_buffer_pool_size innodb_lock_wait_timeout >>> slurmdbd: Accounting storage MYSQL plugin loaded >>> slurmdbd: debug2: ArchiveDir = /tmp >>> slurmdbd: debug2: ArchiveScript = (null) >>> slurmdbd: debug2: AuthAltTypes = (null) >>> slurmdbd: debug2: AuthInfo = (null) >>> slurmdbd: debug2: AuthType = auth/munge >>> slurmdbd: debug2: CommitDelay = 0 >>> slurmdbd: debug2: DbdAddr = localhost >>> slurmdbd: debug2: DbdBackupHost = (null) >>> slurmdbd: debug2: DbdHost = hannibal-hn >>> slurmdbd: debug2: DbdPort = 7032 >>> slurmdbd: debug2: DebugFlags = (null) >>> slurmdbd: debug2: DebugLevel = 6 >>> slurmdbd: debug2: DebugLevelSyslog = 10 >>> slurmdbd: debug2: DefaultQOS = (null) >>> slurmdbd: debug2: LogFile = /var/log/slurmdbd.log >>> slurmdbd: debug2: MessageTimeout = 100 >>> slurmdbd: debug2: Parameters = (null) >>> slurmdbd: debug2: PidFile = /run/slurmdbd.pid >>> slurmdbd: debug2: PluginDir = /usr/lib/x86_64-linux-gnu/slurm-wlm >>> slurmdbd: debug2: PrivateData = none >>> slurmdbd: debug2: PurgeEventAfter = 1 months* >>> slurmdbd: debug2: PurgeJobAfter = 12 months* >>> slurmdbd: debug2: PurgeResvAfter = 1 months* >>> slurmdbd: debug2: PurgeStepAfter = 1 months >>> slurmdbd: debug2: PurgeSuspendAfter = 1 months >>> slurmdbd: debug2: PurgeTXNAfter = 12 months >>> slurmdbd: debug2: PurgeUsageAfter = 24 months >>> slurmdbd: debug2: SlurmUser = root(0) >>> slurmdbd: debug2: StorageBackupHost = (null) >>> slurmdbd: debug2: StorageHost = localhost >>> slurmdbd: debug2: StorageLoc = slurm_acct_db >>> slurmdbd: debug2: StoragePort = 3306 >>> slurmdbd: debug2: StorageType = accounting_storage/mysql >>> slurmdbd: debug2: StorageUser = slurm >>> slurmdbd: debug2: TCPTimeout = 2 >>> slurmdbd: debug2: TrackWCKey = 0 >>> slurmdbd: debug2: TrackSlurmctldDown= 0 >>> slurmdbd: debug2: acct_storage_p_get_connection: request new connection >>> 1 >>> slurmdbd: debug2: Attempting to connect to localhost:3306 >>> slurmdbd: slurmdbd version 19.05.5 started >>> slurmdbd: debug2: running rollup at Thu May 30 13:50:08 2024 >>> slurmdbd: debug2: Everything rolled up >>> >>> >>> It goes like this for some time and then it crashes with this message >>> >>> slurmdbd: Terminate signal (SIGINT or SIGTERM) received >>> slurmdbd: debug: rpc_mgr shutting down >>> >>> >>> On Thu, May 30, 2024 at 8:18 AM mercan <ahmet.mercan(a)uhem.itu.edu.tr> >>> wrote: >>> >>>> Did you try to connect database using mysql command? >>>> >>>> mysql --user=slurm --password=slurmdbpass slurm_acct_db >>>> >>>> C. Ahmet Mercan >>>> >>>> On 30.05.2024 14:48, Radhouane Aniba via slurm-users wrote: >>>> >>>> Thank you Ahmet, >>>> I dont have a firewall active. >>>> And because slurmdbd cannot connect to the database I am not able to >>>> getting it to be activated through systemctl I will share the output for >>>> slurmdbd -D -vvv shortly but overall it is always saying trying to connect >>>> to the db and then retries a couple of times and crashes >>>> >>>> R. >>>> >>>> >>>> >>>> >>>> On Thu, May 30, 2024 at 2:51 AM mercan <ahmet.mercan(a)uhem.itu.edu.tr> >>>> wrote: >>>> >>>>> Hi; >>>>> >>>>> Did you check can you connect db with your conf parameters from >>>>> head-node: >>>>> >>>>> mysql --user=slurm --password=slurmdbpass slurm_acct_db >>>>> >>>>> Also, check and stop firewall and selinux, if they are running. >>>>> >>>>> Last, you can stop slurmdbd, then run run terminal with: >>>>> >>>>> slurmdbd -D -vvv >>>>> >>>>> Regards; >>>>> >>>>> C. Ahmet Mercan >>>>> >>>>> On 30.05.2024 00:05, Radhouane Aniba via slurm-users wrote: >>>>> >>>>> Hi everyone >>>>> I am trying to get slurmdbd to run on my local home server but I am >>>>> really struggling. >>>>> Note : am a novice slurm user >>>>> my slurmdbd always times out even though all the details in the conf >>>>> file are correct >>>>> >>>>> My log looks like this >>>>> >>>>> [2024-05-29T20:51:30.088] Accounting storage MYSQL plugin loaded >>>>> [2024-05-29T20:51:30.088] debug2: ArchiveDir = /tmp >>>>> [2024-05-29T20:51:30.088] debug2: ArchiveScript = (null) >>>>> [2024-05-29T20:51:30.088] debug2: AuthAltTypes = (null) >>>>> [2024-05-29T20:51:30.088] debug2: AuthInfo = (null) >>>>> [2024-05-29T20:51:30.088] debug2: AuthType = auth/munge >>>>> [2024-05-29T20:51:30.088] debug2: CommitDelay = 0 >>>>> [2024-05-29T20:51:30.088] debug2: DbdAddr = localhost >>>>> [2024-05-29T20:51:30.088] debug2: DbdBackupHost = (null) >>>>> [2024-05-29T20:51:30.088] debug2: DbdHost = head-node >>>>> [2024-05-29T20:51:30.088] debug2: DbdPort = 7032 >>>>> [2024-05-29T20:51:30.088] debug2: DebugFlags = (null) >>>>> [2024-05-29T20:51:30.088] debug2: DebugLevel = 6 >>>>> [2024-05-29T20:51:30.088] debug2: DebugLevelSyslog = 10 >>>>> [2024-05-29T20:51:30.088] debug2: DefaultQOS = (null) >>>>> [2024-05-29T20:51:30.088] debug2: LogFile = /var/log/slurmdbd.log >>>>> [2024-05-29T20:51:30.088] debug2: MessageTimeout = 100 >>>>> [2024-05-29T20:51:30.088] debug2: Parameters = (null) >>>>> [2024-05-29T20:51:30.088] debug2: PidFile = /run/slurmdbd.pid >>>>> [2024-05-29T20:51:30.088] debug2: PluginDir = >>>>> /usr/lib/x86_64-linux-gnu/slurm-wlm >>>>> [2024-05-29T20:51:30.088] debug2: PrivateData = none >>>>> [2024-05-29T20:51:30.088] debug2: PurgeEventAfter = 1 months* >>>>> [2024-05-29T20:51:30.088] debug2: PurgeJobAfter = 12 months* >>>>> [2024-05-29T20:51:30.088] debug2: PurgeResvAfter = 1 months* >>>>> [2024-05-29T20:51:30.088] debug2: PurgeStepAfter = 1 months >>>>> [2024-05-29T20:51:30.088] debug2: PurgeSuspendAfter = 1 months >>>>> [2024-05-29T20:51:30.088] debug2: PurgeTXNAfter = 12 months >>>>> [2024-05-29T20:51:30.088] debug2: PurgeUsageAfter = 24 months >>>>> [2024-05-29T20:51:30.088] debug2: SlurmUser = root(0) >>>>> [2024-05-29T20:51:30.089] debug2: StorageBackupHost = (null) >>>>> [2024-05-29T20:51:30.089] debug2: StorageHost = localhost >>>>> [2024-05-29T20:51:30.089] debug2: StorageLoc = slurm_acct_db >>>>> [2024-05-29T20:51:30.089] debug2: StoragePort = 3306 >>>>> [2024-05-29T20:51:30.089] debug2: StorageType = >>>>> accounting_storage/mysql >>>>> [2024-05-29T20:51:30.089] debug2: StorageUser = slurm >>>>> [2024-05-29T20:51:30.089] debug2: TCPTimeout = 2 >>>>> [2024-05-29T20:51:30.089] debug2: TrackWCKey = 0 >>>>> [2024-05-29T20:51:30.089] debug2: TrackSlurmctldDown= 0 >>>>> [2024-05-29T20:51:30.089] debug2: acct_storage_p_get_connection: >>>>> request new connection 1 >>>>> [2024-05-29T20:51:30.089] debug2: Attempting to connect to >>>>> localhost:3306 >>>>> [2024-05-29T20:51:30.090] slurmdbd version 19.05.5 started >>>>> [2024-05-29T20:51:30.090] debug2: running rollup at Wed May 29 >>>>> 20:51:30 2024 >>>>> [2024-05-29T20:51:30.091] debug2: Everything rolled up >>>>> [2024-05-29T20:51:49.673] Terminate signal (SIGINT or SIGTERM) >>>>> received >>>>> [2024-05-29T20:51:49.673] debug: rpc_mgr shutting down >>>>> >>>>> >>>>> >>>>> my config file looks like this >>>>> >>>>> ArchiveEvents=yes >>>>> ArchiveJobs=yes >>>>> ArchiveResvs=yes >>>>> ArchiveSteps=no >>>>> ArchiveSuspend=no >>>>> ArchiveTXN=no >>>>> ArchiveUsage=no >>>>> PurgeEventAfter=1month >>>>> PurgeJobAfter=12month >>>>> PurgeResvAfter=1month >>>>> PurgeStepAfter=1month >>>>> PurgeSuspendAfter=1month >>>>> PurgeTXNAfter=12month >>>>> PurgeUsageAfter=24month >>>>> # Authentication info >>>>> AuthType=auth/munge >>>>> # slurmDBD info >>>>> DbdAddr=localhost >>>>> DbdHost=head-node >>>>> DbdPort=7032 >>>>> SlurmUser=root >>>>> MessageTimeout=100 >>>>> DebugLevel=5 >>>>> #DefaultQOS=normal,standby >>>>> LogFile=/var/log/slurmdbd.log >>>>> PidFile=/run/slurmdbd.pid >>>>> #PrivateData=accounts,users,usage,jobs >>>>> #TrackWCKey=yes >>>>> # >>>>> # Database info >>>>> StorageType=accounting_storage/mysql >>>>> StorageHost=localhost >>>>> StoragePort=3306 >>>>> StoragePass=slurmdbpass >>>>> StorageUser=slurm >>>>> StorageLoc=slurm_acct_db >>>>> I used standard names and passwords to get started and I will change >>>>> later >>>>> >>>>> but everytime I try to start slurmdbd.service it crashes and I have >>>>> that log that I shared with you >>>>> >>>>> I use these versions >>>>> >>>>> slurmdbd -V >>>>> slurm-wlm 19.05.5 >>>>> mysql Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu (x86_64) >>>>> using readline 5.2 >>>>> Everything else Is working properly except I cannot get slurmdbd to >>>>> work and at this point I exhausted all my possible trials :) looking for >>>>> some expert insights :) >>>>> >>>>> >>>>> Any idea what I am doing wrong here ? Also I didn't compile any slurm >>>>> package. I used the binary from apt repos >>>>> >>>>> Any help will be appreciated >>>>> >>>>> Cheers >>>>> >>>>> Rad >>>>> >>>>> -- >>>>> >>>>> >>>>> >>>> >>> >>> -- >>> *Rad Aniba, PhD* >>> >>> >>> >>> -- >>> slurm-users mailing list -- slurm-users(a)lists.schedmd.com >>> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com >>> >> >> >> -- >> *Rad Aniba, PhD* >> >> > > -- > *Rad Aniba, PhD* > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > > > -- *Rad Aniba, PhD*

2 3

Slurm version 24.05.0 is now available
by Marshall Garey 30 May '24

30 May '24

We are pleased to announce the availability of Slurm 24.05.0. To highlight some new features in 24.05: - Isolated Job Step management. Enabled on a job-by-job basis with the --stepmgr option, or globally through SlurmctldParameters=enable_stepmgr. - Federation - Allow for client command operation while SlurmDBD is unavailable. - New MaxTRESRunMinsPerAccount and MaxTRESRunMinsPerUser QOS limits. - New USER_DELETE reservation flag. - New Flags=rebootless option on Features for node_features/helpers which indicates the given feature can be enabled without rebooting the node. - Cloud power management options: New "max_powered_nodes=<limit>" option in SlurmctldParamters, and new SuspendExcNodes=<nodes>:<count> syntax allowing for <count> nodes out of a given node list to be excluded. - StdIn/StdOut/StdErr now stored in SlurmDBD accounting records for batch jobs. - New switch/nvidia_imex plugin for IMEX channel management on NVIDIA systems. - New RestrictedCoresPerGPU option at the Node level, designed to ensure GPU workloads always have access to a certain number of CPUs even when nodes are running non-GPU workloads concurrently. The Slurm documentation has also been updated to the 24.05 release. (Older versions can be found in the archive, linked from the main documentation page.) Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Executing srun -n X where X is greater than total CPU in entire cluster
by Dan Healy 30 May '24

30 May '24

Hi there, SLURM community, I swear I've done this before, but now it's failing on a new cluster I'm deploying. We have 6 compute nodes with 64 cpu each (384 CPU total). When I run `srun -n 500 hostname`, the task gets queued since there's not 500 available CPU. Wasn't there an option that allows for this to be run where the first 384 tasks execute, and then the remaining execute when resources free up? Here's my conf: # Slurm Cgroup Configs used on controllers and workersslurm_cgroup_config: CgroupAutomount: yes ConstrainCores: yes ConstrainRAMSpace: yes ConstrainSwapSpace: yes ConstrainDevices: yes# Slurm conf file settingsslurm_config: AccountingStorageType: "accounting_storage/slurmdbd" AccountingStorageEnforce: "limits" AuthAltTypes: "auth/jwt" ClusterName: "cluster" AccountingStorageHost : "{{ hostvars[groups['controller'][0]].ansible_hostname }}" DefMemPerCPU: 1024 InactiveLimit: 120 JobAcctGatherType: "jobacct_gather/cgroup" JobCompType: "jobcomp/none" MailProg: "/usr/bin/mail" MaxArraySize: 40000 MaxJobCount: 100000 MinJobAge: 3600 ProctrackType: "proctrack/cgroup" ReturnToService: 2 SelectType: "select/cons_tres" SelectTypeParameters: "CR_Core_Memory" SlurmctldTimeout: 30 SlurmctldLogFile: "/var/log/slurm/slurmctld.log" SlurmdLogFile: "/var/log/slurm/slurmd.log" SlurmdSpoolDir: "/var/spool/slurm/d" SlurmUser: "{{ slurm_user.name }}" SrunPortRange: "60000-61000" StateSaveLocation: "/var/spool/slurm/ctld" TaskPlugin: "task/affinity,task/cgroup" UnkillableStepTimeout: 120 -- Thanks, Daniel Healy

2 2

Jobs showing running but not running
by Sushil Mishra 29 May '24

29 May '24

Hi All, I'm managing a cluster with Slurm, consisting of 4 nodes. One of the compute nodes appears to be experiencing issues. While the front node's 'squeue' command indicates that jobs are running, upon connecting to the problematic node, I observe no active processes and GPUs are not being utilized. [sushil@ccbrc ~]$ sinfo -Nel Wed May 29 12:00:08 2024 NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON gag 1 defq* mixed 48 2:24:1 370000 0 1 (null) none gag 1 glycore mixed 48 2:24:1 370000 0 1 (null) none glyco1 1 defq* completing* 128 2:64:1 500000 0 1 (null) none glyco1 1 glycore completing* 128 2:64:1 500000 0 1 (null) none glyco2 1 defq* mixed 128 2:64:1 500000 0 1 (null) none glyco2 1 glycore mixed 128 2:64:1 500000 0 1 (null) none mannose 1 defq* mixed 24 2:12:1 180000 0 1 (null) none mannose 1 glycore mixed 24 2:12:1 180000 0 1 (null) none On glyco1 (affected node!): squeue # gets stuck sudo systemctl restart slurmd # gets stuck I tried the following to clear the jobs stuck in CG state, but any new job appears to be stuck in a 'running' state without actually running. scontrol update nodename=glyco1 state=down reason=cg scontrol update nodename=glyco1 state=resume reason=cg There is no I/O issue in that node, and all file systems are under 30% in use. Any advice on how to resolve this without rebooting the machine? Best, Sushil

3 2

dynamical configuration || meta configuration mgmt
by Heckes, Frank 29 May '24

29 May '24

Hello all, I’m sorry if this has been asked and answered before, but I couldn’t find anything related. Does anyone know whether a framework of sorts exists that allow to change certain SLURM configuration parameters provided some conditions in the batch system’s state are detected and of course are revert if the state became the old one again? (To be more concrete: We like to raise or unset maxjobPU to run as much small jobs as possible to allocate all nodes as soon as certain threshold of free nodes are available and of course some other scenarios) Many thanks in advance. Cheers, -Frank Max-Planck-Institut für Sonnensystemforschung Justus-von-Liebig-Weg 3 D-37077 Göttingen Phone: [+49] 551 – 384 979 320 E-Mail: <mailto:heckes@mps.mpg.de> heckes(a)mps.mpg.de

2 1

Configuring sacct to report state=OUT_OF_MEMORY
by Lee 29 May '24

29 May '24

Hello, *Background :* I am working on a small cluster that is managed by Base Command Manager v10.0 using Slurm 23.02.7 with Ubuntu 22.04.2. I have a small testing script that simply consumes memory and processors. I run my test script, it consumes more memory than allocated by Slurm and as expected it gets killed by OOM killer. In /var/log/slurmd, I see entries like : [2024-05-29T08:53:04.975] Launching batch job 65 for UID 1001 [2024-05-29T08:53:05.016] [65.batch] task/cgroup: _memcg_initialize: job: alloc=10868MB mem.limit=10868MB memsw.limit=10868MB job_swappiness=1 [2024-05-29T08:53:05.016] [65.batch] task/cgroup: _memcg_initialize: step: alloc=10868MB mem.limit=10868MB memsw.limit=10868MB job_swappiness=1 [2024-05-29T08:53:19.530] [65.batch] task/cgroup: task_cgroup_memory_check_oom: StepId=65.batch hit memory+swap limit at least once during execution. This may or may not result in some failure. [2024-05-29T08:53:19.563] [65.batch] done with job Inspecting with sacct, I see : $ sacct -j 65 --format="jobid,jobname,state,exitcode" JobID JobName State ExitCode ------------ ---------- ---------- -------- 65 wrap FAILED 9:0 65.batch batch FAILED 9:0 Based on my previous experience with a RHEL based Slurm cluster, I would expect the state to be listed as OUT_OF_MEMORY and the exitcode to be 0:125. *Question : * 1. How do I configure [slurm,cgroup].conf such that when Slurm kills a job due to exceeding the allocated memory, sacct reports the state as OUT_OF_MEMORY? See below for my configuration : *Configuration:* /etc/default/grub : ~# grep -v "^#" /etc/default/grub GRUB_DEFAULT=0 GRUB_TIMEOUT_STYLE=hidden GRUB_TIMEOUT=10 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="" GRUB_CMDLINE_LINUX="biosdevname=0 cgroup_enable=memory swapaccount=1" GRUB_GFXMODE="1024x768,800x600,auto" GRUB_BACKGROUND="/boot/grub/bcm.png" slurm.conf : ~# grep -v "^#" /cm/shared/apps/slurm/var/etc/bcm10-slurm/slurm.conf SlurmUser=slurm SlurmctldPort=6817 SlurmdPort=6818 AuthType=auth/munge SlurmdSpoolDir=/cm/local/apps/slurm/var/spool SwitchType=switch/none MpiDefault=pmix SlurmctldPidFile=/var/run/slurmctld.pid SlurmdPidFile=/var/run/slurmd.pid ProctrackType=proctrack/cgroup ReturnToService=2 TaskPlugin=task/cgroup InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=300 SlurmdTimeout=300 Waittime=0 SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld SlurmdDebug=info SlurmdLogFile=/var/log/slurmd JobAcctGatherType=jobacct_gather/cgroup JobAcctGatherFrequency=30 AccountingStorageType=accounting_storage/slurmdbd AccountingStorageUser=slurm SlurmctldHost=bcm10-h01 AccountingStorageHost=master NodeName=bcm10-n[01,02] Procs=4 CoresPerSocket=4 RealMemory=15988 SocketsPerBoard=1 ThreadsPerCore=1 Boards=1 MemSpecLimit=5120 Feature=location=local PartitionName="defq" Default=YES MinNodes=1 DefaultTime=UNLIMITED MaxTime=UNLIMITED AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 OverSubscribe=NO PreemptMode=OFF AllowAccounts=ALL AllowQos=ALL Nodes=bcm10-n[01,02] ClusterName=bcm10-slurm SchedulerType=sched/backfill StateSaveLocation=/cm/shared/apps/slurm/var/cm/statesave/bcm10-slurm PrologFlags=Alloc GresTypes=gpu Prolog=/cm/local/apps/cmd/scripts/prolog Epilog=/cm/local/apps/cmd/scripts/epilog SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory cgroup.conf ~# grep -v "^#" /cm/shared/apps/slurm/var/etc/bcm10-slurm/cgroup.conf CgroupMountpoint="/sys/fs/cgroup" CgroupAutomount=no ConstrainCores=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes ConstrainDevices=yes AllowedRamSpace=100.00 AllowedSwapSpace=0.00 MemorySwappiness=1 MaxRAMPercent=100.00 MaxSwapPercent=100.00 MinRAMSpace=30 Best regards, Lee

1 0

sbatch problem
by Mihai Ciubancan 29 May '24

29 May '24

Hello, My name is Mihai and a have an issue with a small GPU cluster manage with slurm 22.05.11. I got 2 different output when I'm trying to find out the name of the nodes(one correct and one wrong). The script is: #!/bin/bash #SBATCH --job-name=test #SBATCH --output=/data/mihai/res.txt #SBATCH --partition=eli #SBATCH --nodes=2 srun echo Running on host: $(hostname) srun hostname srun sleep 15 And the output look like this: cat res.txt Running on host: mihai-x8640 Running on host: mihai-x8640 mihaigpu2 mihai-x8640 As you can see the output of the command 'srun echo Running on host: $(hostname)' is the same, as the jobs was running twice on the same node, while command 'srun hostname' it's giving me the correct output. Do you have any idea why the outputs of the 2 commands are different? Thank you, Mihai

2 6

slurm_bufs_sendto failed
by shaobo liu 29 May '24

29 May '24

slurm 23.11.0 , the slurmd error is as follows，what is the reason for this? [2024-05-29T09:21:32.450] task/affinity: task_p_slurmd_batch_request: task_p_slurmd_batch_request: 32259 [2024-05-29T09:21:32.451] task/affinity: batch_bind: job 32259 CPU input mask for node: 0xFFFFFFFFFFFFFFFF [2024-05-29T09:21:32.451] task/affinity: batch_bind: job 32259 CPU final HW mask for node: 0xFFFFFFFFFFFFFFFF [2024-05-29T09:21:32.451] Launching batch job 32259 for UID 66003 [2024-05-29T09:21:32.868] launch task StepId=32259.0 request from UID:66003 GID:66003 HOST:192.168.87.45 PORT:31104 [2024-05-29T09:21:32.868] task/affinity: lllp_distribution: JobId=32259 implicit auto binding: cores,one_thread, dist 2 [2024-05-29T09:21:32.868] task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic [2024-05-29T09:21:32.868] task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [32259]: mask_cpu,one_thread, 0x0000000000000001,0x0000000100000000,0x0000000000000002,0x0000000200000000,0x0000000000000004,0x0000000400000000,0x0000000000000008,0x0000000800000000,0x0000000000000010,0x0000001000000000,0x0000000000000020,0x0000002000000000,0x0000000000000040,0x0000004000000000,0x0000000000000080,0x0000008000000000,0x0000000000000100,0x0000010000000000,0x0000000000000200,0x0000020000000000,0x0000000000000400,0x0000040000000000,0x0000000000000800,0x0000080000000000,0x0000000000001000,0x0000100000000000,0x0000000000002000,0x0000200000000000,0x0000000000004000,0x0000400000000000,0x0000000000008000,0x0000800000000000,0x0000000000010000,0x0001000000000000,0x0000000000020000,0x0002000000000000,0x0000000000040000,0x0004000000000000,0x0000000000080000,0x0008000000000000,0x0000000000100000,0x0010000000000000,0x0000000000200000,0x0020000000000000,0x0000000000400000,0x0040000000000000,0x0000000000800000,0x0080000000000000,0x0000000001000000,0x0100000000000000,0x0000000002000000,0x0200000000000000,0x0000000004000000,0x0400000000000000,0x0000000008000000,0x0800000000000000,0x0000000010000000,0x1000000000000000,0x0000000020000000,0x2000000000000000,0x0000000040000000,0x4000000000000000,0x0000000080000000,0x8000000000000000 [2024-05-29T09:32:04.541] [32259.batch] done with step [2024-05-29T09:33:31.000] error: slurm_send_node_msg: [socket:[7270627]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:34:02.000] error: slurm_send_node_msg: [socket:[7270629]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:34:03.016] [32259.extern] done with step [2024-05-29T09:34:41.000] error: slurm_send_node_msg: [socket:[7270635]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:35:13.000] error: slurm_send_node_msg: [socket:[7270637]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:35:52.000] error: slurm_send_node_msg: [socket:[7270639]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:36:01.000] error: slurm_send_node_msg: [socket:[7270630]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:36:24.000] error: slurm_send_node_msg: [socket:[7270641]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:37:03.000] error: slurm_send_node_msg: [socket:[7270644]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:37:31.000] [32259.0] error: *** STEP 32259.0 STEPD TERMINATED ON node045 AT 2024-05-29T09:37:30 DUE TO JOB NOT RUNNING *** [2024-05-29T09:37:33.000] [32259.0] stepd_cleanup: done with step (rc[0x0]:No error, cleanup_rc[0xfbb]:Job step not running) [2024-05-29T09:37:35.000] error: slurm_send_node_msg: [socket:[7270646]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:38:14.000] error: slurm_send_node_msg: [socket:[7270649]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:38:46.000] error: slurm_send_node_msg: [socket:[7270651]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:39:25.000] error: slurm_send_node_msg: [socket:[7270657]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error

1 0

Final Call for SLUG Presenters & Early Bird Registration!
by Victoria Hobson 28 May '24

28 May '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information and a high-level schedule can be found here:https://slug24.splashthat.com/ The last day to register at the early bird pricing is this Friday, May 31st. Friday is also the deadline to submit a presentation abstract. We do not intend to extend this deadline. If you are interested in presenting your own usage, developments, site report, tutorial, etc about Slurm, please fill out the following form:https://forms.gle/N7bFo5EzwuTuKkBN7 Notifications of final presentations accepted will go out by Friday, June 14th. -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

slurmdbd archive format
by O'Neal, Doug (NIH/NCI) [C] 28 May '24

28 May '24

My organization needs to access historic job information records for metric reporting and resource forecasting. slurmdbd is archiving only the job information, which should be sufficient for our numbers, but is using the default archive script. In retrospect, this data should have been migrated to a secondary MariaDB instance, but that train has passed. The format of the archive files is not well documented. Does anyone have a program (python/C/whatever) that will read a job_table_archive file and decode it into a parsable structure? Douglas O'Neal, Ph.D. (contractor) Manager, HPC Systems Administration Group, ITOG Frederick National Laboratory for Cancer Research Leidos Biomedical Research, Inc. Phone: 301-228-4656 Email: Douglas.O'Neal(a)nih.gov<mailto:Doug%20O'Neal%20%3cDouglas.O'Neal(a)nih.gov%3e>

2 2

Fwd: Re: Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs.
by Hongyi Zhao 28 May '24

28 May '24

---------- Forwarded message --------- From: Hermann Schwärzler <hermann.schwaerzler(a)uibk.ac.at> Date: Tue, May 28, 2024 at 4:10 PM Subject: Re: [slurm-users] Re: Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs. To: Hongyi Zhao <hongyi.zhao(a)gmail.com> Hi Zhao, On 5/28/24 03:08, Hongyi Zhao wrote: [...] > > What's the complete content of cli_filter.lua and where should I put this file? [...] Below you find the complete content of our cli_filter.lua. It has to be put into the same directory as "slurm.conf". --------------------------------- 8< --------------------------------- -- see https://github.com/SchedMD/slurm/blob/master/etc/cli_filter.lua.example function slurm_cli_pre_submit(options, pack_offset) return slurm.SUCCESS end function slurm_cli_setup_defaults(options, early_pass) -- Make --hint=nomultithread the default behavior -- if users specify an other --hint=XX option then -- it will override the setting done here options['hint'] = 'nomultithread' return slurm.SUCCESS end function slurm_cli_post_submit(offset, job_id, step_id) return slurm.SUCCESS end --------------------------------- >8 --------------------------------- Hopefully this helps... Regards, Hermann -- Assoc. Prof. Hongsheng Zhao <hongyi.zhao(a)gmail.com> Theory and Simulation of Materials Hebei Vocational University of Technology and Engineering No. 473, Quannan West Street, Xindu District, Xingtai, Hebei province

1 0

Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs.
by Hongyi Zhao 27 May '24

27 May '24

Dear Slurm Users, I am experiencing a significant performance discrepancy when running the same VASP job through the Slurm scheduler compared to running it directly with mpirun. I am hoping for some insights or advice on how to resolve this issue. System Information: Slurm Version: 21.08.5 OS: Ubuntu 22.04.4 LTS (Jammy) Job Submission Script: #!/usr/bin/env bash #SBATCH -N 1 #SBATCH -D . #SBATCH --output=%j.out #SBATCH --error=%j.err ##SBATCH --time=2-00:00:00 #SBATCH --ntasks=36 #SBATCH --mem=64G echo '#######################################################' echo "date = $(date)" echo "hostname = $(hostname -s)" echo "pwd = $(pwd)" echo "sbatch = $(which sbatch | xargs realpath -e)" echo "" echo "WORK_DIR = $WORK_DIR" echo "SLURM_SUBMIT_DIR = $SLURM_SUBMIT_DIR" echo "SLURM_JOB_NUM_NODES = $SLURM_JOB_NUM_NODES" echo "SLURM_NTASKS = $SLURM_NTASKS" echo "SLURM_NTASKS_PER_NODE = $SLURM_NTASKS_PER_NODE" echo "SLURM_CPUS_PER_TASK = $SLURM_CPUS_PER_TASK" echo "SLURM_JOBID = $SLURM_JOBID" echo "SLURM_JOB_NODELIST = $SLURM_JOB_NODELIST" echo "SLURM_NNODES = $SLURM_NNODES" echo "SLURMTMPDIR = $SLURMTMPDIR" echo '#######################################################' echo "" module purge > /dev/null 2>&1 module load vasp ulimit -s unlimited mpirun vasp_std Performance Observation: When running the job through Slurm: werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ grep LOOP OUTCAR LOOP: cpu time 14.4893: real time 14.5049 LOOP: cpu time 14.3538: real time 14.3621 LOOP: cpu time 14.3870: real time 14.3568 LOOP: cpu time 15.9722: real time 15.9018 LOOP: cpu time 16.4527: real time 16.4370 LOOP: cpu time 16.7918: real time 16.7781 LOOP: cpu time 16.9797: real time 16.9961 LOOP: cpu time 15.9762: real time 16.0124 LOOP: cpu time 16.8835: real time 16.9008 LOOP: cpu time 15.2828: real time 15.2921 LOOP+: cpu time 176.0917: real time 176.0755 When running the job directly with mpirun: werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ mpirun -n 36 vasp_std werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ grep LOOP OUTCAR LOOP: cpu time 9.0072: real time 9.0074 LOOP: cpu time 9.0515: real time 9.0524 LOOP: cpu time 9.1896: real time 9.1907 LOOP: cpu time 10.1467: real time 10.1479 LOOP: cpu time 10.2691: real time 10.2705 LOOP: cpu time 10.4330: real time 10.4340 LOOP: cpu time 10.9049: real time 10.9055 LOOP: cpu time 9.9718: real time 9.9714 LOOP: cpu time 10.4511: real time 10.4470 LOOP: cpu time 9.4621: real time 9.4584 LOOP+: cpu time 110.0790: real time 110.0739 Could you provide any insights or suggestions on what might be causing this performance issue? Are there any specific configurations or settings in Slurm that I should check or adjust to align the performance more closely with the direct mpirun execution? Thank you for your time and assistance. Best regards, Zhao -- Assoc. Prof. Hongsheng Zhao <hongyi.zhao(a)gmail.com> Theory and Simulation of Materials Hebei Vocational University of Technology and Engineering No. 473, Quannan West Street, Xindu District, Xingtai, Hebei province

4 10

Building Slurm debian package vs building from source
by Arnuld 23 May '24

23 May '24

We have several nodes, most of which have different Linux distributions (distro for short). Controller has a different distro as well. The only common thing between controller and all the does is that all of them ar x86_64. I can install Slurm using package manager on all the machines but this will not work because controller will have a different version of Slurm compared to the nodes (21.08 vs 23.11) If I build from source then I see two solutions: - build a deb package - build a custom package (./configure, make, make install) Building a debian package on the controller and then distributing the binaries on nodes won't work either because that binary will start looking for the shared libraries that it was built for and those don't exist on the nodes. So the only solution I have is to build a static binary using a custom package. Am I correct or is there another solution here?

4 7

Slurm Build Error
by Arnuld 23 May '24

23 May '24

Getting this error when I run "make install": echo >>"lib_ref.lo" /bin/bash ../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o lib_ref.la lib_ref.lo -lpthread -lm -lresolv libtool: link: ar cr .libs/lib_ref.a libtool: link: ranlib .libs/lib_ref.a libtool: link: ( cd ".libs" && rm -f "lib_ref.la" && ln -s "../lib_ref.la" " lib_ref.la" ) /bin/bash ../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -o sacctmgr account_functions.o archive_functions.o association_functions.o config_functions.o cluster_functions.o common.o event_functions.o federation_functions.o file_functions.o instance_functions.o runaway_job_functions.o job_functions.o reservation_functions.o resource_functions.o sacctmgr.o qos_functions.o txn_functions.o user_functions.o wckey_functions.o problem_functions.o tres_function.o -Wl,-rpath=/root/slurm-slurm-23-11-7-1/z/lib/slurm -L../../src/api/.libs -lslurmfull -export-dynamic -lreadline -lhistory lib_ref.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o sacctmgr account_functions.o archive_functions.o association_functions.o config_functions.o cluster_functions.o common.o event_functions.o federation_functions.o file_functions.o instance_functions.o runaway_job_functions.o job_functions.o reservation_functions.o resource_functions.o sacctmgr.o qos_functions.o txn_functions.o user_functions.o wckey_functions.o problem_functions.o tres_function.o -Wl,-rpath=/root/slurm-slurm-23-11-7-1/z/lib/slurm -Wl,--export-dynamic -L../../src/api/.libs /root/slurm-slurm-23-11-7-1/src/api/.libs/libslurmfull.a -lreadline -lhistory ./.libs/lib_ref.a -lpthread -lm -lresolv -pthread /usr/bin/ld: sacctmgr.o: warning: relocation against `_binary_usage_txt_end' in read-only section `.text' /usr/bin/ld: sacctmgr.o: in function `_usage': /root/slurm-slurm-23-11-7-1/src/sacctmgr/sacctmgr.c:926:(.text+0x1f): undefined reference to `_binary_usage_txt_start' /usr/bin/ld: /root/slurm-slurm-23-11-7-1/src/sacctmgr/sacctmgr.c:926:(.text+0x26): undefined reference to `_binary_usage_txt_end' /usr/bin/ld: warning: creating DT_TEXTREL in a PIE collect2: error: ld returned 1 exit status make[3]: *** [Makefile:672: sacctmgr] Error 1 make[3]: Leaving directory '/root/slurm-slurm-23-11-7-1/src/sacctmgr' make[2]: *** [Makefile:520: all-recursive] Error 1 make[2]: Leaving directory '/root/slurm-slurm-23-11-7-1/src' make[1]: *** [Makefile:621: all-recursive] Error 1 make[1]: Leaving directory '/root/slurm-slurm-23-11-7-1' make: *** [Makefile:520: all] Error 2 ----------------------------------------------------------------------------------------- I used these config options: ./configure --enable-debug --enable-pam --disable-sview --disable-shared --with-munge --with-json --with-yaml --with-http-parser --with-pmix --with-lz4 --with-hwloc --with-jwt --with-libcurl --with-freeipmi --with-rdkafka --with-bpf --prefix=/root/slurm-slurm-23-11-7-1/z

1 0

memory high water mark reporting
by Emyr James 22 May '24

22 May '24

Hi, We are trying out slurm having been running grid engine for a long while. In grid engine, the cgroups peak memory and max_rss are generated at the end of a job and recorded. It logs the information from the cgroup hierarchy as well as doing a getrusage call right at the end on the parent pid of the whole job "container" before cleaning up. With slurm it seems that the only way memory is recorded is by the acct gather polling. I am trying to add something in an epilog script to get the memory.peak but It looks like the cgroup hierarchy has been destroyed by the time the epilog is run. Where in the code is the cgroup hierarchy cleared up ? Is there no way to add something in so that the accounting is updated during the job cleanup process so that peak memory usage can be accurately logged ? I can reduce the polling interval from 30s to 5s but don't know if this causes a lot of overhead and in any case this seems to not be a sensible way to get values that should just be determined right at the end by an event rather than using polling. Many thanks, Emyr

5 10

Slurm version 23.11.7 is now available
by Marshall Garey 21 May '24

21 May '24

We are pleased to announce the availability of Slurm version 23.11.7. The 23.11.7 release fixes a few potential crashes in slurmctld when using less common options on job submission, slurmrestd compatibility with auth/slurm, and some additional minor and moderate severity bugs. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Marshall > -- slurmrestd - Correct OpenAPI specification for > 'GET /slurm/v0.0.40/jobs/state' having response as null. > -- Allow running jobs on overlapping partitions if jobs don't specify -s. > -- Fix segfault when requesting a shared gres along with an exclusive > allocation. > -- Fix regression in 23.02 where afternotok and afterok dependencies were > rejected for federated jobs not running on the origin cluster of the > submitting job. > -- slurmctld - Disable job table locking while job state cache is active when > replying to `squeue --only-job-state` or `GET /slurm/v0.0.40/jobs/state`. > -- Fix sanity check when setting tres-per-task on the job allocation as well as > the step. > -- slurmrestd - Fix compatiblity with auth/slurm. > -- Fix issue where TRESRunMins gets off correct value if using > QOS UsageFactor != 1. > -- slurmrestd - Require `user` and `association_condition` fields to be > populated for requests to 'POST /slurmdb/v0.0.40/users_association'. > -- Avoid a slurmctld crash with extra_constraints enabled when a job requests > certain invalid --extra values. > -- `scancel --ctld` and `DELETE /slurm/v0.0/40/jobs` - Fix support for job > array expressions (e.g. 1_[3-5]). Also fix signaling a single pending array > task (e.g. 1_10), which previously signaled the whole array job instead. > -- Fix a possible slurmctld segfault when at some point we failed to create an > external launcher step. > -- Allow the slurmctld to open a connection to the slurmdbd if the first > attempt fails due to a protocol error. > -- mpi/cray_shasta - Fix launch for non-het-steps within a hetjob. > -- sacct - Fix "gpuutil" TRES usage output being incorrect when using --units. > -- Fix a rare deadlock on slurmctld shutdown or reconfigure. > -- Fix issue that only left one thread on each core available when "CPUs=" is > configured to total thread count on multi-threaded hardware and no other > topology info ("Sockets=", "CoresPerSocket", etc.) is configured. > -- Fix the external launcher step not being allocated a VNI when requested. > -- jobcomp/kafka - Fix payload length when producing and sending a message. > -- scrun - Avoid a crash if RunTimeDelete is called before the container > finishes. > -- Save the slurmd's cred_state while reconfiguring to prevent the loss job > credentials.

1 0

SLUG Call for Papers Deadline
by Victoria Hobson 21 May '24

21 May '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information and a high-level schedule can be found here:https://slug24.splashthat.com/ The deadline to submit a presentation abstract is Friday, May 31st. We do not intend to extend this deadline. If you are interested in presenting your own usage, developments, site report, tutorial, etc about Slurm, please fill out the following form:https://forms.gle/N7bFo5EzwuTuKkBN7 Notifications of final presentations accepted will go out by Friday, June 14th. -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

Slurm not allocating correct cgroup cpu ids in srun step (possible bug)
by Ashley Wright 21 May '24

21 May '24

Hi, At our site we have recently upgraded to Slurm 23.11.5 and are having trouble with MPI jobs doing srun inside a sbatch'ed script. The cgroup does not appear to be setup correctly for the srun (step_0). As an example $ cat /sys/fs/cgroup/cpuset/slurm/uid_11000..../job..../cpuset.cpus 0,2-3,68-69,96,98-99,164-165 $ cat /sys/fs/cgroup/cpuset/slurm/uid_11000..../job..../step_0/cpuset.cpus 0,2,68,96,98,164 The sbatch is allocated a range of cpus in the cgroup. However, when step_0 is run, only some of those CPUs are in the group. I have noticed that it is always the range which is missing, ie 2-5 only 2 is included, 3,4,5 are missing. This also only happens if there are multiple groups of cpus in the allocations. ie only 1-12 would be fine, however 1-12,15-20 would result in 1,15 only. The sbatch also seems fine, with step_batch and step_extern being allocated correctly. This causes numerous issues with MPI jobs as they end up overloading cpus. We are running our nodes with threading enabled on the CPUs, and with cgroups and affinity plugins. I have attached our slurm.conf to show our settings. Our /etc/slurm/cgroup.conf is ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes We have turned on logging at debug2 level, but I haven't yet found anything useful. Happy for a suggestion on what to look for. Is anyone able to provide any advice on where to go next to try and identify the issue? Regards, Ashley Wright

1 0

Running slurm on alternate ports
by Alan Stange 20 May '24

20 May '24

Hello all, for testing purposes, we would like to run slurm on ports different from the default values. No problems in setting this up. But how does one tell srun/sbatch/etc what the different port numbers are? I see no command line options to specify a port or an alternate configuration file. Thank you, Alan

2 2

Invalid/incorrect gres.conf syntax
by Gestió Servidors 20 May '24

20 May '24

Hello, I have configured my "gres.conf" in this way: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 node-gpu-1 and node-gpu-2 are two systems with two sockets; node-gpu-3 and node-gpu-4 have only one socket. In my "slurm.conf" I have these lines: AccountingStorageTRES=gres/gpu SelectType=select/cons_tres GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 However, slurmctld reports warning logs about "error syntax in Cores attribute" in gres.conf. Where is the syntax error? Thanks a lot!

1 0

Apply an specific QoS to all users that belongs to an specific account
by Gestió Servidors 20 May '24

20 May '24

Hi, I would like to know if it is possible to apply an specific QoS to all users that belongs to an specific account. For example, I have created some new users "user_XX" and, also, I have created their new accounts in SLURM with "sacctmgr create account name=Test" and "sacctmgr create user name=user_XX DefaultAccount=Test". After it, I have changed default QoS "normal" to a new QoS "minimal" (with some limits) to account "Test" (sacctmgr modify account where name=Test set qos=minimal), but what I have seen is that users "user_XX" that belongs to "Test" account continue in QoS "normal" (QoS by default), so it seems that users have not inherited QoS applied to their account. Is there any way to do this and am I doing something wrong? Thanks.

1 0

srun weirdness
by Dj Merrill 17 May '24

17 May '24

I'm running into a strange issue and I'm hoping another set of brains looking at this might help. I would appreciate any feedback. I have two Slurm Clusters. The first cluster is running Slurm 21.08.8 on Rocky Linux 8.9 machines. The second cluster is running Slurm 23.11.6 on Rocky Linux 9.4 machines. This works perfectly fine on the first cluster: $ srun --mem=32G --pty /bin/bash srun: job 93911 queued and waiting for resources srun: job 93911 has been allocated resources and on the resulting shell on the compute node: $ /mnt/local/ollama/ollama help and the ollama help message appears as expected. However, on the second cluster: $ srun --mem=32G --pty /bin/bash srun: job 3 queued and waiting for resources srun: job 3 has been allocated resources and on the resulting shell on the compute node: $ /mnt/local/ollama/ollama help fatal error: failed to reserve page summary memory runtime stack: runtime.throw({0x1240c66?, 0x154fa39a1008?}) runtime/panic.go:1023 +0x5c fp=0x7ffe6be32648 sp=0x7ffe6be32618 pc=0x4605dc runtime.(*pageAlloc).sysInit(0x127b47e8, 0xf8?) runtime/mpagealloc_64bit.go:81 +0x11c fp=0x7ffe6be326b8 sp=0x7ffe6be32648 pc=0x456b7c runtime.(*pageAlloc).init(0x127b47e8, 0x127b47e0, 0x128d88f8, 0x0) runtime/mpagealloc.go:320 +0x85 fp=0x7ffe6be326e8 sp=0x7ffe6be326b8 pc=0x454565 runtime.(*mheap).init(0x127b47e0) runtime/mheap.go:769 +0x165 fp=0x7ffe6be32720 sp=0x7ffe6be326e8 pc=0x451885 runtime.mallocinit() runtime/malloc.go:454 +0xd7 fp=0x7ffe6be32758 sp=0x7ffe6be32720 pc=0x434f97 runtime.schedinit() runtime/proc.go:785 +0xb7 fp=0x7ffe6be327d0 sp=0x7ffe6be32758 pc=0x464397 runtime.rt0_go() runtime/asm_amd64.s:349 +0x11c fp=0x7ffe6be327d8 sp=0x7ffe6be327d0 pc=0x49421c If I ssh directly to the same node on that second cluster (skipping Slurm entirely), and run the same "/mnt/local/ollama/ollama help" command, it works perfectly fine. My first thought was that it might be related to cgroups. I switched the second cluster from cgroups v2 to v1 and tried again, no difference. I tried disabling cgroups on the second cluster by removing all cgroups references in the slurm.conf file but that also made no difference. My guess is something changed with regards to srun between these two Slurm versions, but I'm not sure what. Any thoughts on what might be happening and/or a way to get this to work on the second cluster? Essentially I need a way to request an interactive shell through Slurm that is associated with the requested resources. Should we be using something other than srun for this? Thank you, -Dj

6 12

Slurm DB upgrade failure behavior
by Yuengling, Philip J. 17 May '24

17 May '24

Hi everyone, I'm writing up some Ansible code to manage Slurm software updates, and I haven't found any documentation about slurmdbd behavior if the mysql/mariadb database doesn't upgrade successfully. What I do know is that if it is sucessful I can expect to see "Conversion done: success!" in the slurmdbd log. This is good, but minor updates do not update the database as far as I know. If the Slurm database cannot upgrade upon an update, does it always shut down with a fatal error? What other behaviors should I look for if there is a failure? Cheers, Phil Y

2 1

Removing safely a node
by Ratnasamy, Fritz 17 May '24

17 May '24

Hi, What is the "official" process to remove nodes safely? I have drained the nodes so jobs are completed and put them in down state after they are completely drained. I edited the slurm.conf file to remove the nodes. After some time, I can see that the nodes were removed from the partition with the command sinfo However, I was told I might need to restart the service slurmctld, do you know if it is necessary? Should I also run scontrol reconfig? Best, *Fritz Ratnasamy* Data Scientist Information Technology

3 2

Best practice for jobs resuming from suspended state
by Paul Jones 16 May '24

16 May '24

Hi, We use PreemptMode and PriorityTier within Slurm to suspend low priority jobs when more urgent work needs to be done. This generally works well, but on occasion resumed jobs fail to restart - which is to say Slurm sets the job status to running but the actual code doesn't recover from being suspended. Technically everything is working as expected, but I wondered if there was any best practice to pass onto users about how to cope with this state? Obviously not a direct Slurm question, but wondered if others had experience with this and any advice on how best to limit the impact? Thanks, Paul --

2 1

Location of Slurm source packages?
by Jeffrey Layton 15 May '24

15 May '24

Good morning, I have an Ubuntu 22.04 server where I installed Slurm from the Ubuntu packages. I now want to install pyxis but it says I need the Slurm sources. In Ubuntu 22.04, is there a package that has the source code? How to download the sources I need from github? Thanks! Jeff

2 2

Re: Location of Slurm source packages?
by Lloyd Brown 15 May '24

15 May '24

Jeff, I'm not sure what version is in the Ubuntu packages, as I don't think they're provided by SchedMD, and I'm having trouble finding the right one on packages.ubuntu.com. Having said that, SchedMD is pretty good about using tags in their github repo (https://github.com/schedmd/slurm), to represent the releases. For example, the "slurm-23-11-6-1" tag corresponds to release 23.11.6. It's pretty straightforward to clone the repo, and do something like "git checkout -b MY_LOCAL_BRANCH_NAME TAG_NAME" to get the version you're after. Lloyd -- Lloyd Brown HPC Systems Administrator Office of Research Computing Brigham Young University http://rc.byu.edu On 5/15/24 08:35, Jeffrey Layton via slurm-users wrote: > Good morning, > > I have an Ubuntu 22.04 server where I installed Slurm from the Ubuntu > packages. I now want to install pyxis but it says I need the Slurm > sources. In Ubuntu 22.04, is there a package that has the source code? > How to download the sources I need from github? > > Thanks! > > Jeff

3 3

Re: Location of Slurm source packages?
by Lloyd Brown 15 May '24

15 May '24

Jeff, Dang. That's really old. I'm not sure I would run one that old, to be honest. Too many missing security fixes and added features. It's never been that hard to do a 'git clone' and the normal configure/make/make install process with slurm. Someone else made me aware of this, in case it's easier: https://slurm.schedmd.com/quickstart_admin.html#debuild Lloyd On 5/15/24 08:57, Jeffrey Layton wrote: > Lloyd, > > Good to hear from you! I was hoping to avoid the use of git but that > may be the only way. The version is 21.08.5. I checked the "old" > packages from SchedMD and they begin part way through 2024 so that > won't work. > > I'm very surprised Ubuntu let a package through without a source > package for it. I'm hoping I'm not seeing the tree through the forest > in finding that package. > > Thanks for the help! > > Jeff > > > On Wed, May 15, 2024 at 10:54 AM Lloyd Brown via slurm-users > <slurm-users(a)lists.schedmd.com> wrote: > > Jeff, > > I'm not sure what version is in the Ubuntu packages, as I don't > think they're provided by SchedMD, and I'm having trouble finding > the right one on packages.ubuntu.com > <http://packages.ubuntu.com>. Having said that, SchedMD is pretty > good about using tags in their github repo > (https://github.com/schedmd/slurm), to represent the releases. > For example, the "slurm-23-11-6-1" tag corresponds to release > 23.11.6. It's pretty straightforward to clone the repo, and do > something like "git checkout -b MY_LOCAL_BRANCH_NAME TAG_NAME" to > get the version you're after. > > > Lloyd > > -- > Lloyd Brown > HPC Systems Administrator > Office of Research Computing > Brigham Young University > http://rc.byu.edu > > On 5/15/24 08:35, Jeffrey Layton via slurm-users wrote: >> Good morning, >> >> I have an Ubuntu 22.04 server where I installed Slurm from the >> Ubuntu packages. I now want to install pyxis but it says I need >> the Slurm sources. In Ubuntu 22.04, is there a package that has >> the source code? How to download the sources I need from github? >> >> Thanks! >> >> Jeff > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > -- Lloyd Brown HPC Systems Administrator Office of Research Computing Brigham Young University http://rc.byu.edu

1 0

Job Invalid Account
by Joe Teumer 15 May '24

15 May '24

We installed slurm 23.11.5 and we are receiving "JobId=n has invalid account" for every sbatch job. We are not using the slurm accounting/user database; we are using uniform UIDs and GIDs across the cluster. The jobs run and complete; can these invalid account errors be ignored or silenced? Job Submission Environment: id joteumer uid=938401109(joteumer) gid=938400513(SPG) groups=938400513(SPG),27(sudo) Slurm Worker Node: id joteumer uid=938401109(joteumer) gid=938400513(SPG) groups=938400513(SPG),27(sudo) slurmctld log: [2024-04-18T09:46:40.000] sched: JobId=18 has invalid account scontrol show job 18 JobId=18 JobName=simplejob.sh UserId=joteumer(938401109) GroupId=SPG(938400513) MCS_label=N/A Priority=1 Nice=0 *Account=(null) *QOS=(null) Submit another sbatch job and update the job to include an Account scontrol update jobid=19 Account=joteumer [2024-04-18T09:56:05.126] _slurm_rpc_submit_batch_job: JobId=19 InitPrio=1 usec=485 [2024-04-18T09:56:06.000] sched: JobId=19 has invalid account [2024-04-18T09:56:17.000] debug: set_job_failed_assoc_qos_ptr: Filling in assoc for JobId=19 Assoc=0 [2024-04-18T09:56:17.000] sched: JobId=19 has invalid account [2024-04-18T09:56:17.588] debug: set_job_failed_assoc_qos_ptr: Filling in assoc for JobId=19 Assoc=0 [2024-04-18T09:56:27.505] _slurm_rpc_update_job: complete JobId=19 uid=0 usec=110 [2024-04-18T09:56:28.000] sched: JobId=19 has invalid account scontrol show job 19 JobId=19 JobName=simplejob.sh UserId=joteumer(938401109) GroupId=SPG(938400513) MCS_label=N/A Priority=1 Nice=0 Account=(null) QOS=(null) JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) 19 SPG simplejob joteumer PENDING 0:00 18:00:00 1 (InvalidAccount)

2 1

Slurm Cleaning Up $XDG_RUNTIME_DIR Before It Should?
by Arnuld 15 May '24

15 May '24

I am using the latest slurm. It runs fine for scripts. But if I give it a container then it kills it as soon as I submit the job. Is slurm cleaning up the $XDG_RUNTIME_DIR before it should? This is the log: [2024-05-15T08:00:35.143] [90.0] debug2: _generate_patterns: StepId=90.0 TaskId=-1 [2024-05-15T08:00:35.143] [90.0] debug3: _get_container_state: command argv[0]=/bin/sh [2024-05-15T08:00:35.143] [90.0] debug3: _get_container_state: command argv[1]=-c [2024-05-15T08:00:35.143] [90.0] debug3: _get_container_state: command argv[2]=crun --rootless=true --root=/run/user/1000/ state slurm2.acog.90.0.-1 [2024-05-15T08:00:35.167] [90.0] debug: _get_container_state: RunTimeQuery rc:256 output:error opening file `/run/user/1000/slurm2.acog.90.0.-1/status`: No such file or directory [2024-05-15T08:00:35.167] [90.0] error: _get_container_state: RunTimeQuery failed rc:256 output:error opening file `/run/user/1000/slurm2.acog.90.0.-1/status`: No such file or directory [2024-05-15T08:00:35.167] [90.0] debug: container already dead [2024-05-15T08:00:35.167] [90.0] debug3: _generate_spooldir: task:0 pattern:%m/oci-job%j-%s/task-%t/ path:/var/spool/slurmd/oci-job90-0/task-0/ [2024-05-15T08:00:35.167] [90.0] debug2: _generate_patterns: StepId=90.0 TaskId=0 [2024-05-15T08:00:35.168] [90.0] debug3: _generate_spooldir: task:-1 pattern:%m/oci-job%j-%s/ path:/var/spool/slurmd/oci-job90-0/ [2024-05-15T08:00:35.168] [90.0] stepd_cleanup: done with step (rc[0x100]:Unknown error 256, cleanup_rc[0x0]:No error) [2024-05-15T08:00:35.275] debug3: in the service_connection [2024-05-15T08:00:35.278] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-05-15T08:00:35.278] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-05-15T08:00:35.278] debug: _rpc_terminate_job: uid = 64030 JobId=90 [2024-05-15T08:00:35.278] debug: credential for job 90 revoked

2 2

Slurm release candidate version 24.05.0rc1 available for testing
by Marshall Garey 14 May '24

14 May '24

We are pleased to announce the availability of Slurm release candidate 24.05.0rc1. To highlight some new features coming in 24.05: - (Optional) isolated Job Step management. Enabled on a job-by-job basis with the --stepmgr option, or globally through SlurmctldParameters=enable_stepmgr. - Federation - Allow for client command operation while SlurmDBD is unavailable. - New MaxTRESRunMinsPerAccount and MaxTRESRunMinsPerUser QOS limits. - New USER_DELETE reservation flag. - New Flags=rebootless option on Features for node_features/helpers which indicates the given feature can be enabled without rebooting the node. - Cloud power management options: New "max_powered_nodes=<limit>" option in SlurmctldParamters, and new SuspendExcNodes=<nodes>:<count> syntax allowing for <count> nodes out of a given node list to be excluded. - StdIn/StdOut/StdErr now stored in SlurmDBD accounting records for batch jobs. - New switch/nvidia_imex plugin for IMEX channel management on NVIDIA systems. - New RestrictedCoresPerGPU option at the Node level, designed to ensure GPU workloads always have access to a certain number of CPUs even when nodes are running non-GPU workloads concurrently. This is the first release candidate of the upcoming 24.05 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 24.05.x version and we will address them before the first production 24.05.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Submitting from an untrusted node
by Rike-Benjamin Schuppner 14 May '24

14 May '24

Hi, If I understand it correctly, the MUNGE and SACK authentication modules naturally require that no-one can get access to the key. This means that we should not use our normal workstations to which our users have physical access to run any jobs, nor could our users use the workstations to submit jobs to the compute nodes. They would have to ssh to a specific submit node and only then could they schedule their jobs. Is there an elegant way to enable job submission from any computer (possibly requiring that users type their password for the submit node – or to their ssh key – at some point)? (All computers/users use the same LDAP server for logins.) Best /rike

2 1

Which "oci.conf" to use?
by Arnuld 14 May '24

14 May '24

I have installed slurm and podman. I have replaced podman's default runtime as per the documentation to "slurm". Documentation says I need to choose one oci.conf: https://slurm.schedmd.com/containers.html#example Which one should I use? runc? crun? nvidia?

1 0

any way to allow interactive jobs or ssh in Slurm 23.02 when node is draining?
by Robert Kudyba 13 May '24

13 May '24

We use Bright Cluster Manager with SLurm 23.02 on RHEL9. I know about pam_slurm_adopt https://slurm.schedmd.com/pam_slurm_adopt.html which does not appear to come by default with the Bright 'cm' package of Slurm. Currently ssh to a node gets: Login not allowed: no running jobs and no WLM allocations We have 8 GPUs on a node so when we drain a node, which can have up to a 5 day job, no new jobs can run. And since we have 20+ TB (yes TB) local drives, researchers have their work and files on them to retrieve. Is there a way to use /etc/security/access.conf to work around this at least temporarily until the reboot and then we can revert? Thanks! Rob

2 2

Announcing Slurm-web v3.0.0, open source web dashboard for Slurm
by Rémi Palancher 13 May '24

13 May '24

Hello Slurm users, Some of you may find interest in the new major version of Slurm-web v3.0.0, an open source web dashboard for Slurm: https://slurm-web.com Slurm-web provides a reactive & responsive web interface to track jobs with intuitive insights and advanced visualizations to monitor status of HPC supercomputers in your organization. The software is released under GPLv3 [1]. This new version is based on official Slurm REST API slurmrestd and adopts modern web technologies to provide many features: - Instant jobs filtering and sorting - Live jobs status update - Advanced visualization of node status with racking topology - Intuitive visualization of QOS and advanced reservations - Multi-clusters support - LDAP authentication - Advanced RBAC permissions management - Transparent caching For the next releases, a roadmap is published with many features ideas [2]. Quick start guide to install: http://docs.rackslab.io/slurm-web/install/quickstart.html RPM and deb packages are published for easy installation and upgrade on all most popular Linux distributions. I hope you will like it! [1] https://github.com/rackslab/Slurm-web [2] https://slurm-web.com/roadmap/ -- Rémi Palancher Rackslab: Open Source Solutions for HPC Operations https://rackslab.io

1 0

srun launched mpi job occasionally core dumps
by Henderson, Brent 10 May '24

10 May '24

Greetings Slurm gurus -- I've been having an issue where very occasionally an srun launched OpenMPI job launched will die during startup within MPI_Init(). E.g. srun -N 8 --ntasks-per-node=1 ./hello_world_mpi. Same binary launched with mpirun does not experience the issue. E.g. mpirun -n 64 -H cn01,... ./hello_world_mpi. The failure rate seems to be in the 0.5% - 1.0% range when using srun for launch. SW stack is self-built with: * Dual socket AMD nodes * RHEL 9.3 base system + tools * Single 100 Gb card per host * hwloc 2.9.3 * pmix 4.2.9 (5.0.2 also tried but continued to see the same issues) * slurm 23.11.6 (started with 23.11.5 - update did not change the behavior) * openmpi 5.0.3 The MPI code is a simple hello_world_mpi.c - anything that goes through startup via srun - does not seem to matter. Application core dump looks like the following regardless of the test running: [cn04:1194785] *** Process received signal *** [cn04:1194785] Signal: Segmentation fault (11) [cn04:1194785] Signal code: Address not mapped (1) [cn04:1194785] Failing at address: 0xe0 [cn04:1194785] [ 0] /lib64/libc.so.6(+0x54db0)[0x7f54e6254db0] [cn04:1194785] [ 1] /share/openmpi/5.0.3/lib/libmpi.so.40(mca_pml_ob1_recv_frag_callback_match+0x7d)[0x7f54e67eab3d] [cn04:1194785] [ 2] /share/openmpi/5.0.3/lib/libopen-pal.so.80(+0xa7d8c)[0x7f54e6566d8c] [cn04:1194785] [ 3] /lib64/libevent_core-2.1.so.7(+0x21b88)[0x7f54e649cb88] [cn04:1194785] [ 4] /lib64/libevent_core-2.1.so.7(event_base_loop+0x577)[0x7f54e649e7a7] [cn04:1194785] [ 5] /share/openmpi/5.0.3/lib/libopen-pal.so.80(+0x222af)[0x7f54e64e12af] [cn04:1194785] [ 6] /share/openmpi/5.0.3/lib/libopen-pal.so.80(opal_progress+0x85)[0x7f54e64e1365] [cn04:1194785] [ 7] /share/openmpi/5.0.3/lib/libmpi.so.40(ompi_mpi_init+0x46d)[0x7f54e663ce7d] [cn04:1194785] [ 8] /share/openmpi/5.0.3/lib/libmpi.so.40(MPI_Init+0x5e)[0x7f54e66711ae] [cn04:1194785] [ 9] /home/brent/bin/ior-3.0.1/ior[0x403780] [cn04:1194785] [10] /lib64/libc.so.6(+0x3feb0)[0x7f54e623feb0] [cn04:1194785] [11] /lib64/libc.so.6(__libc_start_main+0x80)[0x7f54e623ff60] [cn04:1194785] [12] /home/brent/bin/ior-3.0.1/ior[0x4069d5] [cn04:1194785] *** End of error message *** More than one rank can die with the same stacktrace on a node when this happens - I've seen as many as 6. One other interesting note is that if I change my srun command line to include strace (e.g. srun -N 8 --ntasks-per-node=8 strace <strace-options> ./hello_world_mpi) the issue appears to go away. 0 failures in ~2500 runs. Another thing that seems to help is to disabling cgroups in the slurm.conf. After the change, saw 0 failures in >6100 hello_world_mpi runs. The changes in the slurm.conf were - original: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup,task/affinity JobAcctGatherType=jobacct_gather/cgroup Changed ProctrackType=proctrack/linuxproc TaskPlugin=task/affinity JobAcctGatherType=jobacct_gather/linux My cgroup.conf file contains: ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes AllowedRamSpace=95 Curious is anyone has any thoughts on next steps to help figure out what might be going on and how to resolve it. Currently, I'm planning to back down to the 23.02.7 release and see how that goes but open to other suggestions. Thanks, Brent

2 4

Issues with MIG after update to 23.11
by Cumer Cristiano 10 May '24

10 May '24

Dear list, after an update form 22.05 to 23.11 a host where I'm using MIG started to discard the mig devices during the NVML setuop. I can see those lines in the log: slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) where the MIG devices are discarded during the start of the slurmd deamon. this is my gres.conf file cat /etc/slurm/gres.conf ################################################################## # Slurm's Generic Resource (GRES) configuration file ################################################################## AutoDetect=nvml and that still works for hosts on 22.05 Any idea of what is going on? I tried to disable MIG on the upgraded server, and then all the A100 are recognized without issues. Thanks Cristiano slurmd: debug: Log file re-opened slurmd: debug2: hwloc_topology_init slurmd: debug2: hwloc_topology_load slurmd: debug2: hwloc_topology_export_xml slurmd: debug: CPUs:64 Boards:1 Sockets:2 CoresPerSocket:32 ThreadsPerCore:1 slurmd: debug: cgroup/v1: init: Cgroup v1 plugin loaded slurmd: debug2: hwloc_topology_init slurmd: debug2: xcpuinfo_hwloc_topo_load: xml file (/var/lib/slurm/slurmd/hwloc_topo_whole.xml) found slurmd: debug: CPUs:64 Boards:1 Sockets:2 CoresPerSocket:32 ThreadsPerCore:1 slurmd: debug: gres/gpu: init: loaded slurmd: debug: gpu/nvml: init: init: GPU NVML plugin loaded slurmd: debug2: gpu/nvml: _nvml_init: Successfully initialized NVML slurmd: debug: gpu/nvml: _get_system_gpu_list_nvml: Systems Graphics Driver Version: 550.54.15 slurmd: debug: gpu/nvml: _get_system_gpu_list_nvml: NVML Library Version: 12.550.54.15 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVML API Version: 11 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Total CPU count: 64 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device count: 4 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 0: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-808805ee-2ae5-ee6d-14fd-6e63028d4a2a slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:1:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:01:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: -1,0,0,0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 24-31 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 24-31 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: enabled slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG count: 2 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 0, MIG index 0: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-ab1b2cb1-d690-5564-885e-056f23c1c618 slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-ab1b2cb1-d690-5564-885e-056f23c1c618 slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 1 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 12 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 13 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia0,/dev/nvidia-caps/nvidia-cap12,/dev/nvidia-caps/nvidia-cap13 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 0, MIG index 1: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-c43ecbcb-9f3f-5972-8eb9-6a2a09848bd5 slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-c43ecbcb-9f3f-5972-8eb9-6a2a09848bd5 slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 2 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 21 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 22 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia0,/dev/nvidia-caps/nvidia-cap21,/dev/nvidia-caps/nvidia-cap22 slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 1: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-05911075-e3e1-973b-708a-33a77ddd381c slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:65:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:41:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,-1,0,0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia1 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 8-15 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 8-15 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: enabled slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG count: 2 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 1, MIG index 0: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-36b02f58-efa6-5071-9956-510c8a8e705a slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-36b02f58-efa6-5071-9956-510c8a8e705a slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 1 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 147 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 148 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia1,/dev/nvidia-caps/nvidia-cap147,/dev/nvidia-caps/nvidia-cap148 slurmd: debug2: gpu/nvml: _handle_mig: GPU minor 1, MIG index 1: slurmd: debug2: gpu/nvml: _handle_mig: MIG Profile: nvidia_a100_3g.40gb slurmd: debug2: gpu/nvml: _handle_mig: MIG UUID: MIG-c5ca2779-f34d-544b-af52-55ef14aa0af6 slurmd: debug2: gpu/nvml: _handle_mig: UniqueID: MIG-c5ca2779-f34d-544b-af52-55ef14aa0af6 slurmd: debug2: gpu/nvml: _handle_mig: GPU Instance (GI) ID: 2 slurmd: debug2: gpu/nvml: _handle_mig: Compute Instance (CI) ID: 0 slurmd: debug2: gpu/nvml: _handle_mig: GI Minor Number: 156 slurmd: debug2: gpu/nvml: _handle_mig: CI Minor Number: 157 slurmd: debug2: gpu/nvml: _handle_mig: Device Files: /dev/nvidia1,/dev/nvidia-caps/nvidia-cap156,/dev/nvidia-caps/nvidia-cap157 slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 2: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-7b67964f-9217-908f-042f-f37f06cbb1a2 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:129:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:81:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,-1,4 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia2 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 56-63 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 56-63 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 3: slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100-sxm4-80gb slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-87d7526f-6a29-214e-1c24-722a20211263 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:193:0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:C1:00.0 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,4,-1 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia3 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 40-47 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 40-47 slurmd: debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled slurmd: debug2: Possible GPU Memory Frequencies (1): slurmd: debug2: ------------------------------- slurmd: debug2: *1593 MHz [0] slurmd: debug2: Possible GPU Graphics Frequencies (81): slurmd: debug2: --------------------------------- slurmd: debug2: *1410 MHz [0] slurmd: debug2: *1395 MHz [1] slurmd: debug2: ... slurmd: debug2: *810 MHz [40] slurmd: debug2: ... slurmd: debug2: *225 MHz [79] slurmd: debug2: *210 MHz [80] slurmd: gpu/nvml: _get_system_gpu_list_nvml: 4 GPU system device(s) detected slurmd: debug: Gres GPU plugin: Merging configured GRES with system GPUs slurmd: debug2: gres/gpu: _merge_system_gres_conf: gres_list_conf: slurmd: debug2: GRES[gpu] Type:a100_3g.39gb Count:4 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: debug2: GRES[gpu] Type:a100-sxm4-80gb Count:2 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):56-63 Links:0,0,-1,4 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:(null) slurmd: debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):40-47 Links:0,0,4,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: error: Discarding the following config-only GPU due to lack of File specification: slurmd: error: GRES[gpu] Type:a100_3g.39gb Count:1 Cores(64):(null) Links:(null) Flags:HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT File:(null) UniqueId:(null) slurmd: warning: The following autodetected GPUs are being ignored: slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):24-31 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0,/dev/nvidia-caps/nvidia-cap12,/dev/nvidia-caps/nvidia-cap13 UniqueId:MIG-ab1b2cb1-d690-5564-885e-056f23c1c618 slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):24-31 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0,/dev/nvidia-caps/nvidia-cap21,/dev/nvidia-caps/nvidia-cap22 UniqueId:MIG-c43ecbcb-9f3f-5972-8eb9-6a2a09848bd5 slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):8-15 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1,/dev/nvidia-caps/nvidia-cap147,/dev/nvidia-caps/nvidia-cap148 UniqueId:MIG-36b02f58-efa6-5071-9956-510c8a8e705a slurmd: GRES[gpu] Type:nvidia_a100_3g.40gb Count:1 Cores(64):8-15 Links:(null) Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1,/dev/nvidia-caps/nvidia-cap156,/dev/nvidia-caps/nvidia-cap157 UniqueId:MIG-c5ca2779-f34d-544b-af52-55ef14aa0af6 slurmd: debug2: gres/gpu: _merge_system_gres_conf: gres_list_gpu slurmd: debug2: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):56-63 Links:0,0,-1,4 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:(null) slurmd: debug2: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):40-47 Links:0,0,4,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:(null) slurmd: debug: Gres GPU plugin: Final merged GRES list: slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):56-63 Links:0,0,-1,4 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:(null) slurmd: debug: GRES[gpu] Type:a100-sxm4-80gb Count:1 Cores(64):40-47 Links:0,0,4,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:(null) slurmd: Gres Name=gpu Type=a100-sxm4-80gb Count=1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=a100-sxm4-80gb Count=1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: topology/default: init: topology Default plugin loaded slurmd: CPU frequency setting not configured for this node slurmd: debug: Resource spec: No specialized cores configured by default on this node slurmd: debug: Resource spec: Reserved system memory limit not configured for this node slurmd: debug: task/cgroup: init: Tasks containment cgroup plugin loaded slurmd: task/affinity: init: task affinity plugin loaded with CPU mask 0xffffffffffffffff slurmd: debug: spank: opening plugin stack /var/lib/slurm/slurmd/conf-cache/plugstack.conf slurmd: debug: /var/lib/slurm/slurmd/conf-cache/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf" slurmd: cred/munge: init: Munge credential signature plugin loaded slurmd: warning: Core limit is only 0 KB slurmd: slurmd version 23.11.4 started slurmd: debug2: No acct_gather.conf file (/var/lib/slurm/slurmd/conf-cache/acct_gather.conf) slurmd: debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded slurmd: debug: MPI: Loading all types slurmd: debug: mpi/pmix_v4: init: PMIx plugin loaded slurmd: debug: mpi/pmix_v4: init: PMIx plugin loaded slurmd: debug2: No mpi.conf file (/var/lib/slurm/slurmd/conf-cache/mpi.conf) slurmd: slurmd started on Fri, 10 May 2024 14:30:30 +0000 slurmd: CPUs=64 Boards=1 Sockets=2 Cores=32 Threads=1 Memory=1031664 TmpDisk=0 Uptime=2512944 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) ^Cslurmd: got shutdown request slurmd: all threads complete slurmd: debug: mpi/pmix_v4: fini: (null) [0]: mpi_pmix.c:203: (null): call fini() slurmd: debug: mpi/pmix_v4: fini: (null) [0]: mpi_pmix.c:203: (null): call fini() slurmd: debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded slurmd: debug2: acct_gather_profile_startpoll: poll already ended! slurmd: debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded slurmd: debug: task/affinity: fini: task affinity plugin unloaded slurmd: debug: hash/k12: fini: fini: unloading KangarooTwelve hash plugin slurmd: debug: gres/gpu: fini: unloading ^Cslurmd: debug2: gpu/nvml: _nvml_shutdown: Successfully shut down NVML slurmd: debug: gpu/nvml: fini: fini: unloading GPU NVML plugin slurmd: debug2: acct_gather_profile_startpoll: poll already ended! slurmd: debug: cgroup/v1: fini: unloading Cgroup v1 plugin slurmd: cred/munge: fini: Munge credential signature plugin unloaded slurmd: Slurmd shutdown completing

1 0

Trouble Running Slurm C Extension Plugin
by Glen MacLachlan 10 May '24

10 May '24

Hi, We have a plugin in Lua that mostly does what we want but there are features available in the C extension that are not available to lua. For that reason, we are attempting to convert to C using the guidance found here: https://slurm.schedmd.com/job_submit_plugins.html#building. We arrived here because the lua plugins don't seem to stretch enough to cover the use case we were looking at, i.e., branching off of the value of alloc_id or, for that matter, get_sid(). The goal is to disallow interactive allocations (i.e., salloc) on specific partitions while allowing it on others. However, we've run into an issue with our C plugin right out of the gate and I've included a minimal reproducer as an example which is basically a "Hello World" type of test (job_submit_disallow_salloc.c, see attached). *Expectation* What we expect to happen is a sort of hello-world result with a message being written to a /tmp/min_repo.log but that does not occur. It seems that the plugin does not get run at all when jobs are submitted. Jobs still run as expected but the plugin seems to be ignored. *Steps* We compile gcc -fPIC -DHAVE_CONFIG_H -I /modules/source/slurm-23.02.4 -g -O2 -pthread -fno-gcse -Werror -Wall -g -O0 -fno-strict-aliasing -MT job_submit_disallow_salloc.lo -MD -MP -MF .deps/job_submit_disallow_salloc.Tpo -c job_submit_disallow_salloc.c -o .libs/job_submit_disallow_salloc.o mv .deps/job_submit_disallow_salloc.Tpo .deps/job_submit_disallow_salloc.Plo and link gcc -shared -fPIC -DPIC .libs/job_submit_disallow_salloc.o -O2 -pthread -O0 -pthread -Wl,-soname -Wl,job_submit_disallow_salloc.so -o job_submit_disallow_salloc.so Check links after copying to /usr/lib64/slurm: ldd /usr/lib64/slurm/job_submit_disallow_salloc.so linux-vdso.so.1 (0x00007ffe467aa000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1c02095000) libc.so.6 => /lib64/libc.so.6 (0x00007f1c01cd0000) /lib64/ld-linux-x86-64.so.2 (0x00007f1c024b7000) Can someone point out what we are doing incorrectly or how we might troubleshoot this issue? Kindest regards, Glen *Reproducer* The minimal reproducer is basically a "hello world" for C extensions which I've pasted below (I've also attached it for convenience): #include <slurm/slurm.h> #include <slurm/slurm_errno.h> #include <stdio.h> #include "src/slurmctld/slurmctld.h" const char plugin_name[] = "Min Reproducer"; const char plugin_type[] = "job_submit/disallow_salloc"; const uint32_t plugin_version = SLURM_VERSION_NUMBER; extern int job_submit(job_desc_msg_t *job_desc, uint32_t submit_uid, char **err_msg) { FILE *fp; fp = fopen("/tmp/min_repo.log", "w"); fprintf(fp,"Hello!"); fclose(fp); return SLURM_SUCCESS; } int job_modify(job_desc_msg_t *job_desc, job_record_t *job_ptr, uint32_t submit_uid, char **err_msg) { return SLURM_SUCCESS; }

3 2

2025

2024

slurm-users ----- 2025 ----- July 2025 June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users