- slurm-users - lists.schedmd.com

Re: Slurmd enabled crash with CgroupV2
by Josef Dvoracek 11 Apr '24

11 Apr '24

I observe same behavior on slurm 23.11.5 Rocky Linux8.9.. > [root@compute ~]# cat /sys/fs/cgroup/cgroup.subtree_control > memory pids > [root@compute ~]# systemctl disable slurmd > Removed /etc/systemd/system/multi-user.target.wants/slurmd.service. > [root@compute ~]# cat /sys/fs/cgroup/cgroup.subtree_control > cpuset cpu io memory pids > [root@compute ~]# systemctl enable slurmd > Created symlink /etc/systemd/system/multi-user.target.wants/slurmd.service → /usr/lib/systemd/system/slurmd.service. > [root@compute ~]# cat /sys/fs/cgroup/cgroup.subtree_control > cpuset cpu io memory pids over time (i see this thread is ~1 year old, is here better / new understanding of this? cheers josef On 23. 05. 23 12:46, Alan Orth wrote: > I notice the exact same behavior as Tristan. My CentOS Stream 8 system > is in full unified cgroupv2 mode, the slurmd.service has a > "Delegate=Yes" override added to it, and all cgroup stuff is added to > slurm.conf and cgroup.conf, yet slurmd does not start after reboot. I > don't understand what is happening, but I see the exact same behavior > regarding the cgroup subtree_control with disabling / re-enabling slurmd. >

2 3

Jobs of a user are stuck in Completing stage for a long time and cannot cancel them
by archisman.pathak＠nksecurities.com 11 Apr '24

11 Apr '24

We are running a slurm cluster with version `slurm 22.05.8`. One of our users has reported that their jobs have been stuck at the completion stage for a long time. Referring to Slurm Workload Manager - Slurm Troubleshooting Guide we found that indeed the batchhost for the job was removed from the cluster, perhaps without draining it first. How do we cancel/delete the jobs ? * We tried scancel on the batch and individual job ids from both the user and from SlurmUser

4 5

Upgrading nodes
by Steve Berg 11 Apr '24

11 Apr '24

I just finished migrating a few dozen blade servers from torque to slurm. They're all running Alma 8 currently with the slurm that is available from epel. I do want to get it all upgraded to running Alma 9 and the current version of slurm. Got one system set up as the slurmctld system running Alma 9. I grabbed the tar ball and built RPMs for 9.x. Got a few questions about the best path to proceed. Can I use the Alma 9 system to build rpms for Alma 8? I'm sure I can rig up an 8 system to build rpms on but thought I'd see if there was a way to do it on the one 9 system. My plan will be to get the rpms built for 8 and 9, update the slurmctld system to the latest version of slurm, then update all the nodes to the current slurmd version. Once that's done I should be able to reinstall individual nodes to Alma 9 and the same version of slurmd. Am I missing anything in that sequence? I'm fairly confident that the users aren't running any code that will notice the difference between a node running 8 or 9, that should be transparent to them. -- //- Fixer of that which is broke -// //- Home = sberg(a)mississippi.com -// //- Sinners can repent, but stupid is forever. -//

2 1

single node configuration
by Alison Peterson 10 Apr '24

10 Apr '24

Hi everyone, I'm conducting some tests. I've just set up SLURM on the head node and haven't added any compute nodes yet. I'm trying to test it to ensure it's working, but I'm encountering an error: 'Nodes required for the job are DOWN, DRAINED, or reserved for jobs in higher priority partitions. *[stsadmin@head ~]$ squeue* JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 6 lab test_slu stsadmin PD 0:00 1 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions) *[stsadmin@head ~]$ scontrol show job 6* JobId=6 JobName=test_slurm UserId=stsadmin(1000) GroupId=stsadmin(1000) MCS_label=N/A Priority=1 Nice=0 Account=(null) QOS=normal JobState=PENDING Reason=Nodes_required_for_job_are_DOWN,_DRAINED_or_reserved_for_jobs_in_higher_priority_partitions Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=01:00:00 TimeMin=N/A SubmitTime=2024-04-09T10:43:14 EligibleTime=2024-04-09T10:43:14 AccrueTime=2024-04-09T10:43:14 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-04-09T10:43:23 Scheduler=Backfill:* Partition=lab AllocNode:Sid=head:5147 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1-1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=1G,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=1G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=YES Contiguous=0 Licenses=(null) Network=(null) Command=/home/stsadmin/Downloads/test.sh WorkDir=/home/stsadmin StdErr=/home/stsadmin/test_slurm_output.txt StdIn=/dev/null StdOut=/home/stsadmin/test_slurm_output.txt Power= *[stsadmin@head ~]$ scontrol show node head* NodeName=head CoresPerSocket=6 CPUAlloc=0 CPUEfctv=24 CPUTot=24 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=head NodeHostName=head RealMemory=184000 AllocMem=0 FreeMem=N/A Sockets=2 Boards=1 State=DOWN+NOT_RESPONDING ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=lab BootTime=None SlurmdStartTime=None LastBusyTime=2024-04-09T10:42:53 ResumeAfterTime=None CfgTRES=cpu=24,mem=184000M,billing=24 AllocTRES= CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/a ExtSensorsWatts=0 ExtSensorsTemp=n/a Reason=Not responding [slurm@2024-04-09T10:14:10] I will take any advice to guide me in the proper direction, thank you! -- *Alison Peterson* IT Research Support Analyst *Information Technology* apeterson5(a)sdsu.edu <mfarley(a)sdsu.edu> O: 619-594-3364 *San Diego State University | SDSU.edu <http://sdsu.edu/>* 5500 Campanile Drive | San Diego, CA 92182-8080

2 1

Avoiding fragmentation
by Gerhard Strangar 10 Apr '24

10 Apr '24

Hi, I'm trying to figure out how to deal with a mix of few- and many-cpu jobs. By that I mean most jobs use 128 cpus, but sometimes there are jobs with only 16. As soon as that job with only 16 is running, the scheduler splits the next 128 cpu jobs into 96+16 each, instead of assigning a full 128 cpu node to them. Is there a way for the administrator to achieve preferring full nodes? The existence of pack_serial_at_end makes me believe there is not, because that basically is what I needed, apart from my serial jobs using 16 cpus instead of 1. Gerhard

6 5

Nodes required for job are down, drained or reserved
by Alison Peterson 09 Apr '24

09 Apr '24

Hi everyone, I'm conducting some tests. I've just set up SLURM on the head node and haven't added any compute nodes yet. I'm trying to test it to ensure it's working, but I'm encountering an error: 'Nodes required for the job are DOWN, DRAINED, or reserved for jobs in higher priority partitions. Any guidance will be appreciated thank you! -- *Alison Peterson* IT Research Support Analyst *Information Technology* apeterson5(a)sdsu.edu <mfarley(a)sdsu.edu> O: 619-594-3364 *San Diego State University | SDSU.edu <http://sdsu.edu/>* 5500 Campanile Drive | San Diego, CA 92182-8080

2 9

DefCpuPerGPU and multiple partitions
by Ansgar Esztermann-Kirchner 09 Apr '24

09 Apr '24

Hello List, does anyone have experience with DefCpuPerGPU and jobs requesting multiple partitions? I would expect Slurm to select a partition from those requested by the job, then assign CPUs based on that partition's DefCpuPerGPU. But according to my observations, it appears that (at least sometimes), DefCpuPerGPU from another partition (also requested by the job, but not selected to execute it) is used to assign CPUs. I haven't done extensive testing to determine whether there's a pattern to this (e.g. always the first partition is used to calculate the number of CPUs), but I've found a bug report that seems to describe what I'm seeing: https://bugs.schedmd.com/show_bug.cgi?id=15238 Unfortunately, it appears to have been closed as invalid really quickly. I'm on 23.02.5. Best, A. -- Ansgar Esztermann Sysadmin Dep. Theoretical and Computational Biophysics https://www.mpinat.mpg.de/person/11315/3883774

1 0

Elastic Computing: Is it possible to incentivize grouping power_up calls?
by Xaver Stiensmeier 09 Apr '24

09 Apr '24

Dear slurm user list, we make use of elastic cloud computing i.e. node instances are created on demand and are destroyed when they are not used for a certain amount of time. Created instances are set up via Ansible. If more than one instance is requested at the exact same time, Slurm will pass those into the resume script together and one Ansible call will handle all those instances. However, more often than not workflows will request multiple instances within the same second, but not at the exact same time. This leads to multiple resume script calls and therefore to multiple Ansible calls. This will lead to less clear log files, greater CPU consumption by the multiple running Ansible calls and so on. What I am looking for is an option to force Slurm to wait a certain amount and then perform a single resume call for all instances within that time frame (let's say 1 second). Is this somehow possible? Best regards, Xaver

2 2

Slurm User Group 2024 Call for Papers
by Victoria Hobson 08 Apr '24

08 Apr '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information and a high-level schedule can be found here: https://slug24.splashthat.com/ We invite all interested attendees to submit a presentation abstract to be given at SLUG. Presentation content can be in the form of a tutorial, technical presentation or site report. SLUG 2024 is sponsored and organized by the University of Oslo and SchedMD. This international event is open to those who want to: Learn more about Slurm, a highly scalable resource manager and job scheduler - Share their knowledge and experience with other users and administrators - Get detailed information about the latest features and developments - Share requirements and discuss future developments Everyone who wants to present their own usage, developments, site report, or tutorial about Slurm is invited to submit abstract details here: https://forms.gle/N7bFo5EzwuTuKkBN7 Abstracts are due Friday, May 31st and notifications of acceptance will go out by Friday, June 14th. -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

How to reinstall / reconfigure Slurm?
by Shooktija S N 08 Apr '24

08 Apr '24

Hi, I am setting up Slurm on our lab's 3 node cluster and I have run into a problem while adding GPUs (each node has an NVIDIA 4070 ti) as a GRES. There is an error at the 'debug' log level in slurmd.log that says that the GPU is file-less and is being removed from the final GRES list. This error according to some older posts on this forum might be fixed by reinstalling / reconfiguring Slurm with the right flag (the '--with-nvml' flag according to this <https://groups.google.com/g/slurm-users/c/cvGb4JnK8BY> post). Line in /var/log/slurmd.log: [2024-04-03T15:42:02.695] debug: Removing file-less GPU gpu:rtx4070 from final GRES list Does this error require me to either reinstall / reconfigure Slurm? What does 'reconfigure Slurm' mean? I'm about as clueless as a caveman with a smartphone when it comes to Slurm administration and Linux system administration in general. So, if you could, please explain it to me as simply as possible. slurm.conf without comment lines: ClusterName=DlabCluster SlurmctldHost=server1 GresTypes=gpu ProctrackType=proctrack/linuxproc ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=root StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/affinity,task/cgroup InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 SchedulerType=sched/backfill SelectType=select/cons_tres JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=debug2 SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=debug2 SlurmdLogFile=/var/log/slurmd.log NodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:rtx4070:1 PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP gres.conf (only one line): AutoDetect=nvml While installing cuda, I know that nvml has been installed because of this line in /var/log/cuda-installer.log: [INFO]: Installing: cuda-nvml-dev Thanks! PS: I could've added this as a continuation to this post <https://groups.google.com/g/slurm-users/c/p68dkeUoMmA>, but for some reason I do not have permission to post to that group, so here I am starting a new thread.

2 3