- slurm-users - lists.schedmd.com

slurm reservation: how to use more nodes (bigger than NodeCnt) when submitting jobs
by WANG, Hongying 03 May '24

03 May '24

Hi all, I have a question about the reservation. If I create a reservation with 3 nodes (NodeCnt=3), how can I use more than 3 nodes when submitting a job with --reservation=$RES_NAME ? Thanks a lot. Regards, -- WANG, Hongying ClusterTech Limited Advanced I.T. Solutions and Consultancy SINCE 2000 Cloud + High Performance Computing + AI & Analytics + Big Data www.clustertech.com Units 211-213, Lakeside 1, No. 8 Science Park West Avenue, Hong Kong Science Park, N.T. Hong Kong Hong Kong • Beijing • Shanghai • Guangzhou • Shenzhen • Xi'an • Sydney

1 0

scheduling according time requirements
by Dietmar Rieder 30 Apr '24

30 Apr '24

Hi, is it possible to have slurm scheduling jobs automatical according to the "-t" time requirements to a fitting partition? e.g. 3 partitions PartitionName=standard Nodes=c-[01-10] Default=YES MaxTime=04:00:00 DefaultTime=00:10:00 State=UP OverSubscribe=NO PartitionName=medium Nodes=c-[04-08] Default=NO MaxTime=24:00:00 DefaultTime=04:00:00 State=UP OverSubscribe=NO PartitionName=long Nodes=c-[09-10] Default=NO MaxTime=336:00:00 DefaultTime=24:00:00 State=UP OverSubscribe=NO So in the standard partition which is the default we have all nodes and a max time of 4h, in the medium partition we have 4 nodes with a max time of 24h and in the long partition we have 2 nodes with a max time of 336h. I was hoping that if I submit a job with -t 01:00:00 it can be run on any node (standard partition), whereas when specifying -t 05:00:00 or -t 48:00:00 the job will run on the nodes of the medium or long partition respectively. However, my job will not get scheduled at all when -t is greater than 01:00:00 i.e. ]$ srun --cpus-per-task 1 -t 01:00:01 --pty bash srun: Requested partition configuration not available now srun: job 42095 queued and waiting for resources it will wait forever because the standard partition is selected, I was thinking that slurm would automatically switch to the medium partition. Do I misunderstand something there? Or can this be somehow configured. Thanks so much and sorry for the naive question Dietmar

4 10

Trying to Track Down root Usage
by Jason Simms 29 Apr '24

29 Apr '24

Hello all, Each week, I generate an automated report of the top users by CPU hours. This week, for whatever reason the user root accounted for a massive number of hours: -------------------------------------------------------- Login Proper Name Used Account --------------- -------------------- --------- --------- root root 28224 root I've tried to generate a report on what might account for this, but so far I'm stumped. For example, when I run a typical report: sacct -o jobid,jobname,start,end,NNodes,NCPUS,ReqMem,CPUTime,AveRSS,MaxRSS -S 2024-04-22 -E 2024-04-29 --user=root --units=G Nothing comes up (same if I change --user=root to --account=root). I'd appreciate any thoughts as to how to investigate or understand this. It may well be that I'm missing something fundamental. Warmest regards, Jason -- *Jason L. Simms, Ph.D., M.P.H.* Manager of Research Computing Swarthmore College Information Technology Services (610) 328-8102 Schedule a meeting: https://calendly.com/jlsimms

2 2

HPC Principal System Engineer at the Broad
by Paul Edmon 25 Apr '24

25 Apr '24

A friend ask me to pass this along. Figured some folks on this list might be interested. https://broadinstitute.avature.net/en_US/careers/JobDetail/HPC-Principal-Sy… -Paul Edmon-

1 0

Apply an specific QoS to all users that belongs to an specific account
by Gestió Servidors 23 Apr '24

23 Apr '24

Hi, I would like to know if it is possible to apply an specific QoS to all users that belongs to an specific account. For example, I have created some new users "user_XX" and, also, I have created their new accounts in SLURM with "sacctmgr create account name=Test" and "sacctmgr create user name=user_XX DefaultAccount=Test". After it, I have changed default QoS "normal" to a new QoS "minimal" (with some limits) to account "Test" (sacctmgr modify account where name=Test set qos=minimal), but what I have seen is that users "user_XX" that belongs to "Test" account continue in QoS "normal" (QoS by default), so it seems that users have not inherited QoS applied to their account. Is there any way to do this and am I doing something wrong? Thanks. -- [cid:image003.jpg@01DA9569.AE233AA0]<http://www.uab.cat/> Daniel Ruiz Molina Tècnic Mitjà Informàtic Arquitectura de Computadors i Sistemes Operatius Escola d'Enginyeria Edifici Q - Despatx QC/3052 - Carrer de les Sitges Campus de la UAB · 08193 Bellaterra (Cerdanyola del Vallès) · Barcelona · Spain +34 93 581 35 44 www.uab.cat<http://www.uab.cat/> Daniel Ruiz at UAB<https://tinyurl.com/yd95zb8j> [cid:image004.jpg@01DA9569.AE233AA0]<www.linkedin.com/in/daniel-ruiz-molina-50a83b27> Aquest missatge s'adreça exclusivament a la persona destinatària i pot contenir informació privada o confidencial. Si l'heu rebut per error, comuniqueu-nos-ho i destruïu-lo, i tingueu present que no teniu autorització per fer-ne cap ús. Abans d'imprimir aquest missatge penseu en el medi ambient.

1 0

Integrating Slurm with WekaIO
by Jeffrey Layton 19 Apr '24

19 Apr '24

Good afternoon, I'm working on a cluster of NVIDIA DGX A100's that is using BCM 10 (Base Command Manager which is based on Bright Cluster Manager). I ran into an error and only just learned that Slurm and Weka don't get along (presumably because Weka pins their client threads to cores). I read through their documentation here: https://docs.weka.io/best-practice-guides/weka-and-slurm-integration#headin… I through I set everything correctly but when I try to restart the slurm server I get the following: Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: slurmd: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: slurmd: error: fetch_config: DNS SRV lookup failed Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: slurmd: error: _establish_configuration: failed to load configs Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: slurmd: error: slurmd initialization failed Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: error: fetch_config: DNS SRV lookup failed Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: error: _establish_configuration: failed to load configs Apr 19 05:29:39 bcm10-headnode slurmd[3992058]: error: slurmd initialization failed Apr 19 05:29:39 bcm10-headnode systemd[1]: slurmd.service: Main process exited, code=exited, status=1/FAILURE Apr 19 05:29:39 bcm10-headnode systemd[1]: slurmd.service: Failed with result 'exit-code'. Has anyone encountered this? I read this is usually associated with configless Slurm, but I don't know how Slurm is built in BCM. slurm.conf is located in /cm/shared/apps/slurm/var/etc/slurm and this is what I edited. The environment variables for Slurm are set correctly so it points to this slurm.conf file. One thing that I did not do was tell Slurm which cores Weka was using. I can seem to figure out the syntax for this. Can someone share the changes they made to slurm.conf? Thanks! Jeff

3 5

Munge log-file fills up the file system to 100%
by Ole Holm Nielsen 19 Apr '24

19 Apr '24

We have some new AMD EPYC compute nodes with 96 cores/node running RockyLinux 8.9. We've had a number of incidents where the Munge log-file /var/log/munge/munged.log suddenly fills up the root file system, after a while to 100% (tens of GBs), and the node eventually comes to a grinding halt! Wiping munged.log and restarting the node works around the issue. I've tried to track down the symptoms and this is what I found: 1. In munged.log there are infinitely many lines filling up the disk: 2024-04-11 09:59:29 +0200 Info: Suspended new connections while processing backlog 2. The slurmd is not getting any responses from munged, even though we run "munged --num-threads 10". The slurmd.log displays errors like: [2024-04-12T02:05:45.001] error: If munged is up, restart with --num-threads=10 [2024-04-12T02:05:45.001] error: Munge encode failed: Failed to connect to "/var/run/munge/munge.socket.2": Resource temporarily unavailable [2024-04-12T02:05:45.001] error: slurm_buffers_pack_msg: auth_g_create: RESPONSE_ACCT_GATHER_UPDATE has authentication error 3. The /var/log/messages displays the errors from slurmd as well as NetworkManager saying "Too many open files in system". The telltale syslog entry seems to be: Apr 12 02:05:48 e009 kernel: VFS: file-max limit 65536 reached where the limit is confirmed in /proc/sys/fs/file-max. We have never before seen any such errors from Munge. The error may perhaps be triggered by certain user codes (possibly star-ccm+) that might be opening a lot more files on the 96-core nodes than on nodes with a lower core count. My workaround has been to edit the line in /etc/sysctl.conf: fs.file-max = 131072 and update settings by "sysctl -p". We haven't seen any of the Munge errors since! The version of Munge in RockyLinux 8.9 is 0.5.13, but there is a newer version in https://github.com/dun/munge/releases/tag/munge-0.5.16 I can't figure out if 0.5.16 has a fix for the issue seen here? Questions: Have other sites seen the present Munge issue as well? Are there any good recommendations for setting the fs.file-max parameter on Slurm compute nodes? Thanks for sharing your insights, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

4 10

Reserving resources for use by non-slurm stuff
by Shooktija S N 17 Apr '24

17 Apr '24

Hi, I am running Slurm (v22.05.8) on 3 nodes each with the following specs: OS: Proxmox VE 8.1.4 x86_64 (based on Debian 12) CPU: AMD EPYC 7662 (128) GPU: NVIDIA GeForce RTX 4070 Ti Memory: 128 Gb This is /etc/slurm/slurm.conf on all 3 computers without the comment lines: ClusterName=DlabCluster SlurmctldHost=server1 GresTypes=gpu ProctrackType=proctrack/linuxproc ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=root StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/affinity,task/cgroup InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 SchedulerType=sched/backfill SelectType=select/cons_tres JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=debug3 SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=debug3 SlurmdLogFile=/var/log/slurmd.log NodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1 PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP I want to reserve a few cores and a few gigs of RAM for use only by the OS which cannot be accessed by jobs being managed by Slurm. What configuration do I need to do to achieve this? Is it possible to reserve in a similar fashion a 'percent' of the GPU which Slurm cannot exceed so that the OS has some GPU resources? Is it possible to have these configs be different for each of the 3 nodes? Thanks!

3 2

Inconsistencies in CPU time Reporting by sreport and sacct Tools
by KK 17 Apr '24

17 Apr '24

I wish to ascertain the CPU core time utilized by user dj1 and dj. I have tested with sreport cluster UserUtilizationByAccount, sreport job SizesByAccount, and sacct. It appears that sreport cluster UserUtilizationByAccount displays the total core hours used by the entire account, rather than the individual user's cpu time. Here are the specifics: Users dj and dj1 are both under the account mehpc. In 2024-04-12 ~ 2024-04-15, dj1 used approximately 10 minutes of core time, while dj used about 4 minutes. However, "sreport Cluster UserUtilizationByAccount user=dj1 start=2024-04-12 end=2024-04-15" shows 14 minutes of usage. Similarly, "sreport job SizesByAccount Users=dj start=2024-04-12 end=2024-04-15" hows about 14 minutes. Using "sreport job SizesByAccount Users=dj1 start=2024-04-12 end=2024-04-15" or "sacct -u dj1 -S 2024-04-12 -E 2024-04-15 -o "jobid,partition,account,user,alloccpus,cputimeraw,state,workdir%60" -X |awk 'BEGIN{total=0}{total+=$6}END{print total}'" yields the accurate values, which are around 10 minutes for dj1. Attachment are the details.

1 0

Association limit problem
by Gestió Servidors 17 Apr '24

17 Apr '24

Hello, I'm doing some test with "associations" with "sacctmgr". I have created three users (user_1, user_2 and user_3). For each of these users, I have created an association: [root@myserver log]# sacctmgr show user user_1 --associations User Def Acct Admin Cluster Account Partition Share Priority MaxJobs MaxNodes MaxCPUs MaxSubmit MaxWall MaxCPUMins QOS Def QOS ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ------- -------- -------- --------- ----------- ----------- -------------------- --------- user_1 test None q50004 test aolin.q 1 4 2 10 normal user_1 test None q50004 test cuda-staf+ 1 4 2 10 normal [root@myserver log]# sacctmgr show user user_2 --associations User Def Acct Admin Cluster Account Partition Share Priority MaxJobs MaxNodes MaxCPUs MaxSubmit MaxWall MaxCPUMins QOS Def QOS ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ------- -------- -------- --------- ----------- ----------- -------------------- --------- user_2 test None q50004 test cuda-int.q 1 4 normal [root@myserver log]# sacctmgr show user user_3 --associations User Def Acct Admin Cluster Account Partition Share Priority MaxJobs MaxNodes MaxCPUs MaxSubmit MaxWall MaxCPUMins QOS Def QOS ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ------- -------- -------- --------- ----------- ----------- -------------------- --------- user_3 test None q50004 test research.q 1 2 1 normal user_3 test None q50004 test xeon.q 1 2 1 normal All users belong to "Test" account: [root@myserver log]# sacctmgr show account test --association Account Descr Org Cluster ParentName User Share Priority GrpJobs GrpNodes GrpCPUs GrpMem GrpSubmit GrpWall GrpCPUMins MaxJobs MaxNodes MaxCPUs MaxSubmit MaxWall MaxCPUMins QOS Def QOS ---------- -------------------- -------------------- ---------- ---------- ---------- --------- ---------- ------- -------- -------- ------- --------- ----------- ----------- ------- -------- -------- --------- ----------- ----------- -------------------- --------- test test test q50004 root 1 normal test test test q50004 user_1 1 4 2 10 normal test test test q50004 user_1 1 4 2 10 normal test test test q50004 user_2 1 4 normal test test test q50004 user_3 1 2 1 normal test test test q50004 user_3 1 2 1 normal When I submit with "user_1", all tests are running fine: some jobs are queued and executed and some jobs are rejected because of the limits. However, with users "user_2" and "user_3" I can't submit any job. All jobs are rejected with these messages: 11168 research. test user_3 PENDING 0:00 2024-04-17T12:53:21 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11173 research. test user_3 PENDING 0:00 2024-04-17T13:06:02 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11174 research. test user_3 PENDING 0:00 2024-04-17T13:06:16 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11176 research. test user_3 PENDING 0:00 2024-04-17T13:07:23 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11180 research. test user_3 PENDING 0:00 2024-04-17T13:08:45 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) For example, user "user_3" are trying to submit in this way (test.sh script only is a simple "sleep 50": sbatch -p aolin.q -N 2 ./test.sh --> sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified sbatch -p aolin.q -N 1 ./test.sh --> sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified sbatch -p research.q -N 1 ./test.sh --> submitted but not running --> nodelist(reason)= (AssocMaxCpuPerJobLimit) -> WHY??? sbatch -p research.q -N 1 -n 1 ./test.sh --> submitted but not running --> nodelist(reason)= (AssocMaxCpuPerJobLimit) --> WHY??? sbatch -p xeon.q -N 1 -n 1 ./test.sh --> submitted and running!! [root@myserver log]# squeue JOBID PARTITION NAME USER STATE TIME SUBMIT_TIME START_TIME NODE CPUS OVER_S TRES_PER_NODE NODELIST(REASON) DEPENDENCY REQ_NODES NODELIST 11202 research. test user_3 PENDING 0:00 2024-04-17T13:33:31 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11200 research. test user_3 PENDING 0:00 2024-04-17T13:33:17 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11212 xeon.q test user_3 RUNNING 0:18 2024-04-17T13:36:10 2024-04-17T13:36:10 1 1 OK N/A aolin-cpu-1 (null) aolin-cpu-1 Why? What am I doing wrong? Where is the limit that I am not seeing? Thanks a lot!

1 0

2025

2024

slurm-users ----- 2025 ----- July 2025 June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users