- slurm-users - lists.schedmd.com

Slurm hanging behavior
by Richard Yang 21 Aug '24

21 Aug '24

Hello Slurm community, We are using slurm as the system to deploy training jobs on a large gpu cluster, but encounter a strange behavior. As new comers, we wonder if this is a known behavior. Below is some more info: * We are running a relatively older version 22.0.5 * At relatively higher load, we encountered hanging. It is particularly puzzling in the following sense: assume we have nodelist1 with 6 hosts and nodelist2 with 7 hosts. We run simple ‘hostname’. Deploying on nodelist1 alone or nodelusr2 alone will be fine, but with all 13 hosts, the debug messages show that the execution hang after showing that the last task done. It then hangs for exactly 180 seconds. Does anyone know the potential issue? We sure be happy to post more config details or debug messages. Thank you so much! Richard

1 0

Access to --constraint=<arg> in Lua cli_filter?
by Kevin Buckley 20 Aug '24

20 Aug '24

If I supply a --constraint=<arg> option to an sbatch/salloc/srun, does the arg appear inside any object that a Lua CLI Filter could access? I've tried this basic check if is_unset(options['constraint']) then slurm_errorf('constraint is unset ') end and seen that that object is, indeed, unset. Kevin

2 2

Unable to run sequential jobs simultaneously on the same node
by Arko Roy 20 Aug '24

20 Aug '24

I want to run 50 sequential jobs (essentially 50 copies of the same code with different input parameters) on a particular node. However, as soon as one of the jobs gets executed, the other 49 jobs get killed immediately with exit code 9. The jobs are not interacting and are strictly parallel. However, if the 50 jobs run on 50 different nodes, it runs successfully. Can anyone please help with possible fixes? I see a discussion almost along the similar lines in https://groups.google.com/g/slurm-users/c/I1T6GWcLjt4 But could not get the final solution. -- Arko Roy Assistant Professor School of Physical Sciences Indian Institute of Technology Mandi Kamand, Mandi Himachal Pradesh - 175 005, India Email: arko(a)iitmandi.ac.in Web: https://faculty.iitmandi.ac.in/~arko/

5 7

sreport syntax for TRES/GPU usage
by Robert Kudyba 17 Aug '24

17 Aug '24

In a 25 node heterogeneous cluster with 4 different types of GPUs, to get granular to see which GPUs were used most over a time period we have to set AccountingStorageTRES to something like: AccountingStorageTRES=gres/gpu,gres/gpu:rtx8000,gres/gpu:v100s,gres/gpu:a40,gres/gpu:a100 Unfortunately it's currently at: AccountingStorageTRES=gres/gpu At least all nodes have the same GPU within each node. What are some good options to sreport to get details on usage over a year, e.g., percentage of CPU vs GPU, which partitions/accounts used the most GPUs, etc. From this example: sreport -tminper -t Percent cluster utilization --tres="cpu,gres/gpu" start=2023-07-01 -------------------------------------------------------------------------------- Cluster Utilization 2023-07-01T00:00:00 - 2024-08-15T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Reserved Reported --------- -------------- ----------- ---------- -------- ----------- ---------- ----------- cluster cpu 43.81% 2.87% 0.00% 48.35% 4.97% 99.86% cluster gres/gpu 50.36% 3.59% 0.00% 46.05% 0.00% 100.38% Is that showing that 50% of all jobs were run with GPUs? How do we read the Idle column? Why does Reported show > 100% for gres?

1 0

Upgrade compute node to 24.05.2
by Sid Young 15 Aug '24

15 Aug '24

G'Day all, I've been upgrading cmy cluster from 20.11.0 in small steps to get to 24.05.2. Currently 1 have all nodes on 23.02.8, the controller on 24.05.2 and a single test node on 24.05.2. All are Centos 7.9 (upgrade to Oracle Linux 8.10 is Phase 2 of the upgrades). When I check the slurmd status on the test node I get: [root@hpc-dev-01 24.05.2]# systemctl status slurmd ● slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2024-08-15 10:45:15 AEST; 24s ago Main PID: 46391 (slurmd) Tasks: 1 Memory: 1.2M CGroup: /system.slice/slurmd.service └─46391 /usr/sbin/slurmd --systemd Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Considering each NUMA node as a socket Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Node reconfigured socket/core boundaries SocketsPerBoard=4:8(hw) CoresPerSocket=16:8(hw) Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Considering each NUMA node as a socket Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: slurmd version 24.05.2 started Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: *plugin_load_from_file: Incompatible Slurm plugin /usr/lib64/slurm/mpi_none.so version (23.02.8)* Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: error: Couldn't load specified plugin name for mpi/none: Incompatible plugin version Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: error: MPI: Cannot create context for mpi/none Aug 15 10:45:15 hpc-dev-01 systemd[1]: Started Slurm node daemon. Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: slurmd started on Thu, 15 Aug 2024 10:45:15 +1000 Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: CPUs=64 Boards=1 Sockets=8 Cores=8 Threads=1 Memory=257778 TmpDisk=15998 Uptime=2898769 CPUSpecL...ve=(null) Hint: Some lines were ellipsized, use -l to show in full. [root@hpc-dev-01 24.05.2]# We don't use MPI (life science workloads)... should I remove the file? If it is version 23.02.8 then doesn't 24.05.2 have that plugin built in? There are no references to mpi i the slurm.conf file Sid

2 1

Seeking Commercial SLURM Subscription Provider
by John Joseph 14 Aug '24

14 Aug '24

Dear All, Good morning. We successfully implemented a 4-node SLURM cluster with shared storage using GlusterFS and were able to run COMSOL programs on it. After this learning experience, we've determined that it would be beneficial to switch to a commercial SLURM subscription for better support. We are currently seeking a solution provider who can offer support based on a commercial subscription. I would like to reach out to the group for recommendations or advice on how we can avail these services commercially. Thank you.Joseph John

2 1

Annoying canonical question about converting SLURM_JOB_NODELIST to a host list for mpirun
by Jeffrey Layton 12 Aug '24

12 Aug '24

Good afternoon, I know this question has been asked a million times, but what is the canonical way to convert the list of nodes for a job that is container in a Slurm variable, I use SLURM_JOB_NODELIST, to a host list appropriate for mpirun in OpenMPI (perhaps MPICH as well)? Before anyone says, compile OpenMPI with Slurm, I can't change the Slurm installation. I have a script that does the conversion on a single node, but when I try a cluster that does not include the single node, I get an error: scontrol: error: host list is empty The line in the script corresponding to this is, list=$(scontrol show hostname $SLURM_NODELIST) I've tried using the env variable SLURM_JOB_NODELIST and I get the same error message. Thanks! Jeff

3 8

FairShare if there's only one account?
by Drucker, Daniel 11 Aug '24

11 Aug '24

Simple question: Does FairShare still work if every user is under one account? E.g.: $ sacctmgr show assoc format=Account,User Account User ---------- ---------- root root root mic mic asmith mic bsmith mic csmith mic djones mic ejones mic frubble Will it divide time up fairly between the users? I have: PriorityType=priority/multifactor PriorityFavorSmall=YES PriorityWeightAge=50000 PriorityWeightFairshare=100000 PriorityWeightJobSize=0 PriorityWeightQOS=0 In 21.08.8. -- Daniel M. Drucker, Ph.D. Director of IT, MGB Imaging at Belmont McLean Hospital, a Harvard Medical School Affiliate The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

5 31

Jobs distribution over CPUs
by Rafał Lalik 10 Aug '24

10 Aug '24

Hi, I have a very simple computing farm on a single PC with AMD Ryzen 7950X (2x16 cores). I have configured my slurm to use up to 25 CPUs: NodeName=palmer CPUs=25 RealMemory=40000 State=UNKNOWN # Boards=1 SocketsPerBoard=1 CoresPerSocket=16 ThreadsPerCore=2 PartitionName=main Nodes=ALL Default=YES MaxTime=INFINITE State=UP But using htop I see that with all 25 jobs running I use max 16 cores. It seems that like * 6 jobs are using 100% of CPU * 20 jobs use 50% CPU each I use slurm 24.05.2, but I am pretty sure in the past when using I think one of 22.x version, the jobs distribution was like 25 CPUs for 25 jobs with 100% each. Is there anything I can do in the configuration of my farm to have better utilisation of CPU cores? Right now like half of them is not really used. Regards, Rafał

1 0

The issue in the distribution of job
by Sundaram Kumaran 09 Aug '24

09 Aug '24

Dear All, May I have your suggestion in my issue facing, While the job is launched using "salloc -N4--mem 4000 -p active" I find the job is running in the one compute node and the other 3 machines are free, I don`t find the job is distributed evenly, May I have your suggestion, I do squeue /scontrol to find the job distribution and it displays the 4 machines but when I check on the respective machines I don`t find the job running only one machine takes the whole node, Is there any issue in my conf file or what needs to be done, May I have your suggestion pls. FYI, salloc -N4 --mem 4000 -p active [cid:image001.png@01DAE8B4.215FBA50] While using TOP, I find only Debussy is used heavily, I don`t find my job is evenly distributed, May I have your guidance pls. [cid:image002.jpg@01DAEA98.2573DE10] Regards, KumaranS This e-mail and any attachments are only for the use of the intended recipient and may contain material that is confidential, privileged and/or protected by the Official Secrets Act. If you are not the intended recipient, please delete it or notify the sender immediately. Please do not copy or use it for any purpose or disclose the contents to any other person.

2 1

2025

2024