- slurm-users - lists.schedmd.com

DefCpuPerGPU and multiple partitions
by Ansgar Esztermann-Kirchner 09 Apr '24

09 Apr '24

Hello List, does anyone have experience with DefCpuPerGPU and jobs requesting multiple partitions? I would expect Slurm to select a partition from those requested by the job, then assign CPUs based on that partition's DefCpuPerGPU. But according to my observations, it appears that (at least sometimes), DefCpuPerGPU from another partition (also requested by the job, but not selected to execute it) is used to assign CPUs. I haven't done extensive testing to determine whether there's a pattern to this (e.g. always the first partition is used to calculate the number of CPUs), but I've found a bug report that seems to describe what I'm seeing: https://bugs.schedmd.com/show_bug.cgi?id=15238 Unfortunately, it appears to have been closed as invalid really quickly. I'm on 23.02.5. Best, A. -- Ansgar Esztermann Sysadmin Dep. Theoretical and Computational Biophysics https://www.mpinat.mpg.de/person/11315/3883774

1 0

Elastic Computing: Is it possible to incentivize grouping power_up calls?
by Xaver Stiensmeier 09 Apr '24

09 Apr '24

Dear slurm user list, we make use of elastic cloud computing i.e. node instances are created on demand and are destroyed when they are not used for a certain amount of time. Created instances are set up via Ansible. If more than one instance is requested at the exact same time, Slurm will pass those into the resume script together and one Ansible call will handle all those instances. However, more often than not workflows will request multiple instances within the same second, but not at the exact same time. This leads to multiple resume script calls and therefore to multiple Ansible calls. This will lead to less clear log files, greater CPU consumption by the multiple running Ansible calls and so on. What I am looking for is an option to force Slurm to wait a certain amount and then perform a single resume call for all instances within that time frame (let's say 1 second). Is this somehow possible? Best regards, Xaver

2 2

Slurm User Group 2024 Call for Papers
by Victoria Hobson 08 Apr '24

08 Apr '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information and a high-level schedule can be found here: https://slug24.splashthat.com/ We invite all interested attendees to submit a presentation abstract to be given at SLUG. Presentation content can be in the form of a tutorial, technical presentation or site report. SLUG 2024 is sponsored and organized by the University of Oslo and SchedMD. This international event is open to those who want to: Learn more about Slurm, a highly scalable resource manager and job scheduler - Share their knowledge and experience with other users and administrators - Get detailed information about the latest features and developments - Share requirements and discuss future developments Everyone who wants to present their own usage, developments, site report, or tutorial about Slurm is invited to submit abstract details here: https://forms.gle/N7bFo5EzwuTuKkBN7 Abstracts are due Friday, May 31st and notifications of acceptance will go out by Friday, June 14th. -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

How to reinstall / reconfigure Slurm?
by Shooktija S N 08 Apr '24

08 Apr '24

Hi, I am setting up Slurm on our lab's 3 node cluster and I have run into a problem while adding GPUs (each node has an NVIDIA 4070 ti) as a GRES. There is an error at the 'debug' log level in slurmd.log that says that the GPU is file-less and is being removed from the final GRES list. This error according to some older posts on this forum might be fixed by reinstalling / reconfiguring Slurm with the right flag (the '--with-nvml' flag according to this <https://groups.google.com/g/slurm-users/c/cvGb4JnK8BY> post). Line in /var/log/slurmd.log: [2024-04-03T15:42:02.695] debug: Removing file-less GPU gpu:rtx4070 from final GRES list Does this error require me to either reinstall / reconfigure Slurm? What does 'reconfigure Slurm' mean? I'm about as clueless as a caveman with a smartphone when it comes to Slurm administration and Linux system administration in general. So, if you could, please explain it to me as simply as possible. slurm.conf without comment lines: ClusterName=DlabCluster SlurmctldHost=server1 GresTypes=gpu ProctrackType=proctrack/linuxproc ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=root StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/affinity,task/cgroup InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 SchedulerType=sched/backfill SelectType=select/cons_tres JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=debug2 SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=debug2 SlurmdLogFile=/var/log/slurmd.log NodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:rtx4070:1 PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP gres.conf (only one line): AutoDetect=nvml While installing cuda, I know that nvml has been installed because of this line in /var/log/cuda-installer.log: [INFO]: Installing: cuda-nvml-dev Thanks! PS: I could've added this as a continuation to this post <https://groups.google.com/g/slurm-users/c/p68dkeUoMmA>, but for some reason I do not have permission to post to that group, so here I am starting a new thread.

2 3

No SLURM_SHARDS_ON_NODE in prolog/epilog
by Thomas Green - Staff in University IT, Research Technologies / Staff Technoleg Gwybodaeth, Technolegau Ymchwil 05 Apr '24

05 Apr '24

Hi, I have found that SLURM_SHARDS_ON_NODE is not an environment variable in prolog or epilog. Is ther ea reason for this? I thought it would be handy to know if shards have been requested in a prolog/epilog. Tom -- Thomas Green Senior Programmer ARCCA, Redwood Building, King Edward VII Avenue, Cardiff, CF10 3NB Tel: +44 (0)29 208 79269 Fax: +44 (0)29 208 70734 Email: greent10(a)cardiff.ac.uk<mailto:greent10@cardiff.ac.uk> Web: http://www.cardiff.ac.uk/arcca Thomas Green Uwch Raglennydd ARCCA, Adeilad Redwood, King Edward VII Avenue, Caerdydd, CF10 3NB Ffôn: +44 (0)29 208 79269 Ffacs: +44 (0)29 208 70734 E-bost: greent10(a)caerdydd.ac.uk<mailto:greent10@caerdydd.ac.uk> Gwefan: http://www.caerdydd.ac.uk/arcca

1 0

Suggestions for Partition/QoS configuration
by thomas.hartmann＠th-ht.de 04 Apr '24

04 Apr '24

Hi, we're testing possible slurm configurations on a test system right now. Eventually, it is going to serve ~1000 users. We're going to have some users who are going to run lots of short jobs (a couple of minutes to ~4h) and some users that run jobs that are going to run for days or weeks. I want to avoid a situation in which a group of users basically saturates the whole cluster with jobs that run for a week or two and nobody could run any short jobs anymore. I also would like to favor short jobs, because they make the whole cluster feel more dynamic and agile for everybody. On the other hand, I would like to make the most of the ressources, i.e. when nobody is sending short jobs, long jobs could run on all the nodes. My idea was to basically have three partitions: 1. PartitionName=short MaxTime=04:00:00 State=UP Nodes=node[01-99] PriorityTier=100 2. PartitionName=long_safe MaxTime=14-00:00:00 State=UP Nodes=node[01-50] PriorityTier=100 3. PartitionName=long_preempt MaxTime=14-00:00:00 State=UP Nodes=nodes[01-99] PriorityTier=40 PreemptMode=requeue and then use the JobSubmitPlugin "all_partitions" so that all jobs get submitted to all partitions by default. This way, a short job ends up in the `short` partition and is able to use all nodes. A long job ends up using the `long_safe` partition until for the first 50 nodes. These jobs are not going to be preempted. Remaining long jobs use the `long_preempt` queue. So they run on the remaining nodes as long as there are no higher prio short (or long) jobs in the queue. So, the cluster could be saturated with long running jobs but if short jobs are submitted and the user has a high enough fair share, some of the long jobs would get preempted and the short ones would run. This scenario works fine.... BUT the long jobs seem to be playing pingpong on the `long_preempt` partition because as soon as they run, they stop accruing AGE priority unlike still queued jobs. As soon as a queued job, albeit by the same user, "overtakes" a running one, it preempts the running one, stops accruing age and so on.... So, is there maybe a cleverer way to do this? Thanks a lot! Thomas

4 4

Re: [EXT] Re: SLURM configuration help
by Renfro, Michael 04 Apr '24

04 Apr '24

Yep, from your scontrol show node output: CfgTRES=cpu=64,mem=2052077M,billing=64 AllocTRES=cpu=1,mem=2052077M The running job (77) has allocated 1 CPU and all the memory on the node. That’s probably due to the partition using the default DefMemPerCPU value [1], which is unlimited. Since all our nodes are shared, and our workloads vary widely, we set our DefMemPerCPU value to something considerably lower than mem_in_node/cores_in_node . That way, most jobs will leave some memory available by default, and other jobs can use that extra memory as long as CPUs are available. [1] https://slurm.schedmd.com/slurm.conf.html#OPT_DefMemPerCPU From: Alison Peterson <apeterson5(a)sdsu.edu> Date: Thursday, April 4, 2024 at 11:58 AM To: Renfro, Michael <Renfro(a)tntech.edu> Subject: Re: [EXT] Re: [slurm-users] SLURM configuration help External Email Warning This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests. ________________________________ Here is the info: sma@cusco:/data/work/sma-scratch/tohoku_wOcean$ scontrol show node cusco NodeName=cusco Arch=x86_64 CoresPerSocket=32 CPUAlloc=1 CPUTot=64 CPULoad=0.02 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:4 NodeAddr=cusco NodeHostName=cusco Version=19.05.5 OS=Linux 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 RealMemory=2052077 AllocMem=2052077 FreeMem=1995947 Sockets=2 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=mainpart BootTime=2024-03-01T17:06:26 SlurmdStartTime=2024-03-01T17:06:53 CfgTRES=cpu=64,mem=2052077M,billing=64 AllocTRES=cpu=1,mem=2052077M CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s sma@cusco:/data/work/sma-scratch/tohoku_wOcean$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 78 mainpart CF1090_w sma PD 0:00 1 (Resources) 77 mainpart CF0000_w sma R 0:26 1 cusco sma@cusco:/data/work/sma-scratch/tohoku_wOcean$ scontrol show job 78 JobId=78 JobName=CF1090_wOcean500m.shell UserId=sma(1008) GroupId=myfault(1001) MCS_label=N/A Priority=4294901720 Nice=0 Account=(null) QOS=(null) JobState=PENDING Reason=Resources Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=01:00:00 TimeMin=N/A SubmitTime=2024-04-04T09:55:34 EligibleTime=2024-04-04T09:55:34 AccrueTime=2024-04-04T09:55:34 StartTime=2024-04-04T10:55:28 EndTime=2024-04-04T11:55:28 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-04-04T09:55:58 Partition=mainpart AllocNode:Sid=newcusco:2450574 ReqNodeList=(null) ExcNodeList=(null) NodeList=(null) SchedNodeList=cusco NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=YES Contiguous=0 Licenses=(null) Network=(null) Command=/data/work/sma-scratch/tohoku_wOcean/CF1090_wOcean500m.shell WorkDir=/data/work/sma-scratch/tohoku_wOcean StdErr=/data/work/sma-scratch/tohoku_wOcean/slurm-78.out StdIn=/dev/null StdOut=/data/work/sma-scratch/tohoku_wOcean/slurm-78.out Power= On Thu, Apr 4, 2024 at 8:57 AM Renfro, Michael <Renfro(a)tntech.edu<mailto:Renfro@tntech.edu>> wrote: What does “scontrol show node cusco” and “scontrol show job PENDING_JOB_ID” show? On one job we currently have that’s pending due to Resources, that job has requested 90 CPUs and 180 GB of memory as seen in its ReqTRES= value, but the node it wants to run on only has 37 CPUs available (seen by comparing its CfgTRES= and AllocTRES= values). From: Alison Peterson via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Date: Thursday, April 4, 2024 at 10:43 AM To: slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Subject: [slurm-users] SLURM configuration help External Email Warning This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests. ________________________________ I am writing to seek assistance with a critical issue on our single-node system managed by Slurm. Our jobs are queued and marked as awaiting resources, but they are not starting despite seeming availability. I'm new with SLURM and my only experience was a class on installing it so I have no experience, running it or using it. Issue Summary: Main Problem: Jobs submitted only one run and the second says NODELIST(REASON) (Resources). I've checked that our single node has enough RAM (2TB) and CPU's (64) available. # COMPUTE NODES NodeName=cusco CPUs=64 Sockets=2 CoresPerSocket=32 ThreadsPerCore=1 RealMemory=2052077 Gres=gpu:1,gpu:1,gpu:1,gpu:1 PartitionName=mainpart Default=YES MinNodes=1 DefaultTime=00:60:00 MaxTime=UNLIMITED AllowAccounts=ALL Nodes=ALL State=UP OverSubscribe=Force System Details: We have a single-node setup with Slurm as the workload manager. The node appears to have sufficient resources for the queued jobs. Troubleshooting Performed: Configuration Checks: I have verified all Slurm configurations and the system's resource availability, which should not be limiting job execution. Service Status: The Slurm daemon slurmdbd is active and running without any reported issues. System resource monitoring shows no shortages that would prevent job initiation. Any guidance and help will be deeply appreciated! -- Alison Peterson IT Research Support Analyst Information Technology apeterson5(a)sdsu.edu<mailto:mfarley@sdsu.edu> O: 619-594-3364 San Diego State University | SDSU.edu<http://sdsu.edu/> 5500 Campanile Drive | San Diego, CA 92182-8080 Error! Filename not specified. -- Alison Peterson IT Research Support Analyst Information Technology apeterson5(a)sdsu.edu<mailto:mfarley@sdsu.edu> O: 619-594-3364 San Diego State University | SDSU.edu<http://sdsu.edu/> 5500 Campanile Drive | San Diego, CA 92182-8080 [Image removed by sender.]

2 1

SLURM configuration help
by Alison Peterson 04 Apr '24

04 Apr '24

I am writing to seek assistance with a critical issue on our single-node system managed by Slurm. Our jobs are queued and marked as awaiting resources, but they are not starting despite seeming availability. I'm new with SLURM and my only experience was a class on installing it so I have no experience, running it or using it. Issue Summary: Main Problem: Jobs submitted only one run and the second says *NODELIST(REASON) (Resources*). I've checked that our single node has enough RAM (2TB) and CPU's (64) available. # COMPUTE NODES NodeName=cusco CPUs=64 Sockets=2 CoresPerSocket=32 ThreadsPerCore=1 RealMemory=2052077 Gres=gpu:1,gpu:1,gpu:1,gpu:1 PartitionName=mainpart Default=YES MinNodes=1 DefaultTime=00:60:00 MaxTime=UNLIMITED AllowAccounts=ALL Nodes=ALL State=UP OverSubscribe=Force System Details: We have a single-node setup with Slurm as the workload manager. The node appears to have sufficient resources for the queued jobs. Troubleshooting Performed: Configuration Checks: I have verified all Slurm configurations and the system's resource availability, which should not be limiting job execution. Service Status: The Slurm daemon slurmdbd is active and running without any reported issues. System resource monitoring shows no shortages that would prevent job initiation. Any guidance and help will be deeply appreciated! -- *Alison Peterson* IT Research Support Analyst *Information Technology* apeterson5(a)sdsu.edu <mfarley(a)sdsu.edu> O: 619-594-3364 *San Diego State University | SDSU.edu <http://sdsu.edu/>* 5500 Campanile Drive | San Diego, CA 92182-8080

2 1

Slurm 23.11 - Unknown system variable 'wsrep_on'
by Russell Jones 04 Apr '24

04 Apr '24

Hi all, I am working on upgrading a Slurm cluster from 20 -> 23. I was successfully able to upgrade to 22, however now that I am trying to go from 22 to 23, starting slurmdbd results in the following error being logged: error: mysql_query failed: 1193 Unknown system variable 'wsrep_on' When trying to start slurmctld, I get: [2024-04-02T15:09:52.439] Couldn't find tres gres/gpumem in the database, creating. [2024-04-02T15:09:52.439] Couldn't find tres gres/gpuutil in the database, creating. [2024-04-02T15:09:52.440] fatal: Problem adding tres to the database, can't continue until database is able to make new tres Any ideas what could be causing these errors? Is MariaDB 5.5 still officially supported?

2 2

scrun: Failed to run the container due to GID mapping configuration
by Toshiki Sonoda (Fujitsu) 04 Apr '24

04 Apr '24

Dear All, We set up scrun (slurm 23.11.5) integrated with rootless podman, referring to the official documentation. https://slurm.schedmd.com/containers.html#podman-scrun https://slurm.schedmd.com/scrun.html#SECTION_Example-%3CB%3Escrun.lua%3C/B%… However, runc/crun prints the error message and fails to run the container. Looking at the error message from runc, it seems that there is a problem with the GID mapping configuration. - runc [hoge@slm-master hoge]$ podman run centos echo "Hello World" time="2024-04-03T02:39:36Z" level=error msg="runc run failed: cannot specify gid=5 mount option for rootless container: User namespaces enabled, but no group mapping found." srun: error: slm-worker: task 0: Exited with exit code 1 - crun [hoge@slm-master hoge]$ podman run centos echo "Hello World" srun: error: slm-worker: task 0: Exited with exit code 1 mount `devpts` to `dev/pts`: Invalid argument When we removed the process of linux.gidMappings configuration from scrun.lua, runc failed to run the container due to the different error, but crun succeeded. - scrun.lua (The process of linux.gidMappings configuration) ... snip ... -- Provide default group map as root if one not provided -- mappings fail with build??? if (true or config["linux"]["gidMappings"] == nil) then config["linux"]["gidMappings"] = {{containerID=process_group_id, hostID=math.floor(group_id), size=1}} end ... snip ... - runc [hoge@slm-master hoge]$ podman run centos echo "Hello World" time="2024-04-03T02:37:18Z" level=error msg="runc run failed: User namespaces enabled, but no gid mappings found." srun: error: slm-worker: task 0: Exited with exit code 1 - crun [hoge@slm-master hoge]$ podman run centos echo "Hello World" Hello World Therefore, we would like to know the following two questions. - The reason for the failure to run the container - Required or recommended configuration to run the container Best Regards, Toshiki Sonoda

2 1