- slurm-users - lists.schedmd.com

slurm.conf
by LEROY Christine 208562 18 Jan '24

18 Jan '24

Hello all, Is there an env variable in SLURM to tell where the slurm.conf is? We would like to have on the same client node, 2 type of possible submissions to address 2 different cluster. Thanks in advance, Christine

4 3

What happens if GPU GRES exceeding number of GPUs per node
by Purwanto, Wirawan 18 Jan '24

18 Jan '24

Hi, In my HPC center, I found a SLURM job that was submitted with --gres=gpu:6 whereas the cluster has only four GPUs per node each. It is a parallel job. Here are some relevant field printout: AllocCPUS 30 AllocGRES gpu:6 AllocTRES billing=30,cpu=30,gres/gpu=6,node=3 CPUTime 1-01:23:00 CPUTimeRAW 91380 Elapsed 00:50:46 JobID 20073 JobIDRaw 20073 JobName simple_cuda NCPUS 30 NGPUS 6.0 What happened in this case? This job was asking for 3 nodes, 10 core per node. When the user specified “--gres=gpu:6”, does this mean six GPUs for the entire job, or six GPUs per node? Per the description in https://slurm.schedmd.com/gres.html#Running_Jobs, it says: gres is “Generic resources required per node”. So it is illogical to request six GPUs per node. So what happened? Did SLURM quietly ignore the request and grant just one, or grant the max number (4)? Because apparently the job ran without error. Wirawan Purwanto Computational Scientist, HPC Group Information Technology Services Old Dominion University Norfolk, VA 23529

2 1

Strict GrpTRESMins limit
by Kamil Wilczek 17 Jan '24

17 Jan '24

Dear All, I tried to implement a strict limit on the GrpTRESMins for each user. The effect I'm trying to achieve is that after the limit of GPU minutes is reached, no new jobs can be run. No decay, no automatic resource replenishment. After the limit on GPU minutes is reached, each user should ask for more minutes. But despite exceeding the limits users *can* run new jobs. * When I'm adding a user to the cluster I set: sacctmgr --immediate add user name=... ... QOS=2gpu2d GrpTRESMins=gres/gpu=20000 * In the "slurm.conf" ("safe" means limits and associations are automatically set). Storage is MariaDB with SlurmDBD: GresTypes=gpu AccountingStorageTRES=gres/gpu AccountingStorageEnforce=qos,safe # This disables GPU minutes replenishing. PriorityDecayHalfLife=0 PriorityUsageResetPeriod=NONE But when I look at a user's account info and usage, you can see that the limits are not enforced. Account User Partition QOS GrpTRESMins ---------- ---------------- ------------ ------------ -------------------- redacted redacted a6000 2gpu2d gres/gpu=10000 -------------------------------------------------------------------------------- Top 1 Users 2024-01-05T00:00:00 - 2024-01-17T19:59:59 (1108800 secs) Usage reported in TRES Minutes -------------------------------------------------------------------------------- Login Used TRES Name ------------ -------- ---------------- redacted 184311 gres/gpu redacted 1558558 cpu Could someone explain, where could the problem be? Am I missing something? Apparently yes :) Kind regards -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

1 0

Suspend/Resume request limit
by 김종록 17 Jan '24

17 Jan '24

2 1

preemptable queue
by Davide DelVento 12 Jan '24

12 Jan '24

I would like to add a preemptable queue to our cluster. Actually I already have. We simply want jobs submitted to that queue be preempted if there are no resources available for jobs in other (high priority) queues. Conceptually very simple, no conditionals, no choices, just what I wrote. However it does not work as desired. This is the relevant part: grep -i Preemp /opt/slurm/slurm.conf #PreemptType = preempt/partition_prio PartitionName=regular DefMemPerCPU=4580 Default=True Nodes=node[01-12] State=UP PreemptMode=off PriorityTier=200 PartitionName=All DefMemPerCPU=4580 Nodes=node[01-36] State=UP PreemptMode=off PriorityTier=500 PartitionName=lowpriority DefMemPerCPU=4580 Nodes=node[01-36] State=UP PreemptMode=cancel PriorityTier=100 That PreemptType setting (now commented) fully breaks slurm, everything refuses to run with errors like $ squeue squeue: error: PreemptType and PreemptMode values incompatible squeue: fatal: Unable to process configuration file If I understand correctly the documentation at https://slurm.schedmd.com/preempt.html that is because preemption cannot cancel jobs based on partition priority, which (if true) is really unfortunate. I understand that allowing cross-partition time-slicing could be tricky and so I understand why that isn't allowed, but cancelling? Anyway, I have to questions: 1) is that correct and so should I avoid using either partition priority or cancelling? 2) is there an easy way to trick slurm into requeing and then have those jobs cancelled instead? 3) I guess the cleanest option would be to implement QoS, but I've never done it and we don't really need it for anything else other than this. The documentation looks complicated, but is it? The great Ole's website is unavailable at the moment... Thanks!!

2 4

Re: [slurm-users] sacct --name --status filtering
by Drucker, Daniel 11 Jan '24

11 Jan '24

Yes, that makes sense. Thank you! The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

sacct --name --status filtering
by Drucker, Daniel 10 Jan '24

10 Jan '24

What am I misunderstanding about how sacct filtering works here? I would have expected the second command to show the exact same results as the first. [root@mickey ddrucker]# sacct --starttime $(date -d "7 days ago" +"%Y-%m-%d") -X --format JobID,JobName,State,Elapsed --name zsh JobID JobName State Elapsed ------------ ---------- ---------- ---------- 257713 zsh COMPLETED 00:01:02 257714 zsh COMPLETED 00:04:01 257715 zsh COMPLETED 00:03:01 257716 zsh COMPLETED 00:03:01 [root@mickey ddrucker]# sacct --starttime $(date -d "7 days ago" +"%Y-%m-%d") -X --format JobID,JobName,State,Elapsed --name zsh --state COMPLETED JobID JobName State Elapsed ------------ ---------- ---------- ---------- [root@mickey ddrucker]# sinfo --version slurm 21.08.8-2 -- Daniel M. Drucker, Ph.D. Director of IT, MGB Imaging at Belmont McLean Hospital, a Harvard Medical School Affiliate The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

3 2

Re: [slurm-users] sacct --name --status filtering
by Drucker, Daniel 10 Jan '24

10 Jan '24

> All I can say is that this has to do with --starttime and that you have to read the manual really carefully about how they interact, including when you have --endtime set. It’s a bit fiddly and annoying, IMO, and I can never quite remember how it works. Oh, I think I understand. --starttime actually behaves differently when --state is present: If states are given with the '-s' option then only jobs in this state at this time will be returned. So is there a way to do what I want? I want to see jobs which - started later than 7 days ago - whose state is COMPLETED Surely that's possible without resorting to grep? Daniel The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

Cleanup of old clusters in database
by Jeffrey R. Lang 10 Jan '24

10 Jan '24

We have shuttered two clusters and need to remove them from the database. To do this, do we remove the table spaces associated with the cluster names from the Slurm database? Thanks, Jeff

1 0

Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error (Timony, Mick)
by Craig Stark 10 Jan '24

10 Jan '24

All good ideas Mick – - I've restarted slurmd on all nodes – no effect - Ran this on all nodes: #!/bin/bash uname -n id slurm id 59999 scontrol show config | grep SlurmUser All show slurm being that 59999 user. - The firewalld already has the internal network interface being used set to the trusted zone I get a bit more info out of setting the slurmctld to debug level, but I'm not sure what to make of it TBH. I'm not sure what "_handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474)" is trying to tell me. Jan 10 10:48:26 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:31 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:35 kirby slurmctld[461138]: slurmctld: debug: accounting_storage/slurmdbd: _handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474): DBD_SEND_MULT_MSG message from invalid uid Jan 10 10:48:36 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:41 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:46 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:46 kirby slurmctld[461138]: slurmctld: debug: sched/backfill: _attempt_backfill: beginning Jan 10 10:48:46 kirby slurmctld[461138]: slurmctld: debug: sched/backfill: _attempt_backfill: no jobs to backfill Jan 10 10:48:51 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:53 kirby slurmctld[461138]: slurmctld: debug: accounting_storage/slurmdbd: _handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474): DBD_SEND_MULT_MSG message from invalid uid Jan 10 10:48:56 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:01 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:06 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:11 kirby slurmctld[461138]: slurmctld: debug: accounting_storage/slurmdbd: _handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474): DBD_SEND_MULT_MSG message from invalid uid Jan 10 10:49:11 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:16 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:17 kirby slurmctld[461138]: slurmctld: debug: sched: Running job scheduler for full queue. Jan 10 10:49:21 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid A bit more info / another possible clue. While "sacctmgr list Account" or "sacctmgr list user" shows expected account groups and users, "sreport user top start=12/1/23" and "sreport cluster utilization start=12/1/23" both report empty tables. Craig Stark, Ph.D. Professor, Department of Neurobiology and Behavior Director, Facility for Imaging and Brain Research (FIBRE) Director, Campus Center for Neuroimaging (CCNI) School of Biological Sciences, University of California, Irvine cestark(a)uci.edu<mailto:cestark@uci.edu>

1 0

Beginner admin question: Prioritization within a partition based on time limit
by Kenneth Chiu 09 Jan '24

09 Jan '24

I'm just learning about slurm. I understand that different different partitions can be prioritized separately, and can have different max time limits. I was wondering whether or not there was a way to have a finer-grained prioritization based on the time limit specified by a job, within a single partition. Or perhaps this is already happening by default? Would the backfill scheduler be best for this?

2 1

job_container/tmpfs and srun.
by Phill Harvey-Smith 09 Jan '24

09 Jan '24

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

1 0

Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error
by Craig Stark 08 Jan '24

08 Jan '24

This ticket with SchedMD implies it's a munged issue: https://urldefense.com/v3/__https://bugs.schedmd.com/show_bug.cgi?id=1293__… Is the munge daemon running on all systems? If it is, are all servers running a network time daemon such chronyd or ntpd and the time is in sync on all hosts? Thanks Mick, munge is seemingly running on all systems (systemctl status munge). I do get a warning about the munge file changing on disk, but I'm pretty sure that's from warewulf sync'ing files every minute. A sha256sum on the munge.key file on the compute nodes and host node says they're the same, so I think I can put that aside. The management node runs chrony and the compute nodes sync to the management node. [root@kirby uber]# chronyc tracking Reference ID : 4A06A849 (t2.time.gq1.yahoo.com) Stratum : 3 Ref time (UTC) : Mon Jan 08 22:26:44 2024 System time : 0.000032525 seconds slow of NTP time Last offset : -0.000021390 seconds RMS offset : 0.000055729 seconds Frequency : 38.797 ppm slow Residual freq : +0.001 ppm Skew : 0.018 ppm Root delay : 0.033342984 seconds Root dispersion : 0.000524800 seconds Update interval : 256.8 seconds Leap status : Normal vs [root@sonic01 ~]# chronyc tracking Reference ID : C0A80102 (warewulf) Stratum : 4 Ref time (UTC) : Mon Jan 08 22:31:02 2024 System time : 0.000000120 seconds slow of NTP time Last offset : -0.000000092 seconds RMS offset : 0.000014737 seconds Frequency : 47.495 ppm slow Residual freq : +0.000 ppm Skew : 0.066 ppm Root delay : 0.033458963 seconds Root dispersion : 0.000283949 seconds Update interval : 64.2 seconds Leap status : Normal So, the compute node is talking to the host and the host is talking to generic NTP sources. "date" shows the same time on the compute nodes

1 0

Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error
by Craig Stark 08 Jan '24

08 Jan '24

3rd time trying to get this to come through to the list - hopefully this time works. I've been running SLURM for several years now, but in setting it up on a new cluster, I'm hitting a recurring issue. I'm using a MariaDB and configured it just as I had in my several-year-ago setup and in the docs. There's a "slurm" user (59999) on the OS (Rocky 9), that's on all the nodes, and I've added the slurm@localhost as instructed (grant all on slurm_acct_db.* TO 'slurm'@'localhost' identified by 'PASSWORD'). But, I keep getting things like this: ``` Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: DBD_SEND_MULT_MSG message from invalid uid 59999 Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: Processing last message from connection 7(192.168.1.2) uid(59999) Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: CONN:7 DBD_REGISTER_CTLD message from invalid uid 59999 Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: CONN:7 Security violation, DBD_REGISTER_CTLD Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: Processing last message from connection 7(192.168.1.2) uid(59999) ``` I'm a total SQL noob, but can at least verify that the user is in there: MariaDB [(none)]> SELECT User, Host, Password FROM mysql.user; +-------------+-----------+-------------------------------------------+ | User | Host | Password | +-------------+-----------+-------------------------------------------+ | mariadb.sys | localhost | | | root | localhost | invalid | | mysql | localhost | invalid | | slurm | localhost | *D6665ECF4F3CB12BCA836117F7727B6D0B78D644 | +-------------+-----------+-------------------------------------------+ 4 rows in set (0.002 sec) Any thoughts as to where I might look to fix this? Craig

2 1

Multifactor fair-share with single account
by Kamil Wilczek 08 Jan '24

08 Jan '24

Dear All, I have a question regarding the fair-share factor of the multifactor priority algorithm. My current understanding is that the fair-share makes sure that different *accounts* have a fair share of the computational power. But what if my organisation structure is flat and I have only one account where all my user reside. Is fair-share algorithm working in this situation -- does it take into account users (associations) from this single account, and tries to assing a fair-factor to each user? Or each user from this account have the same fair-factor at each iteration? And what if I have, say 3 accounts, but I do not wan't to calculate fair-share between accounts, but between all associations from all 3 accounts? In other words, is there a fair-share factor for users/associations instead of accounts? Kind regards -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

4 6

Tool for profiling resource usage by slurm jobs
by Nicolas Granger 08 Jan '24

08 Jan '24

Hi all, Happy new year everyone! I've been looking for a simple tool that reports how much resources are actually consumed by a job to help my colleagues and I adjust job requirements. I could not find such a tool, or the ones mentioned on this ML were not easy to install and use, so I have written a new one: https://github.com/CEA-LIST/sprofile It's a simple python script which parses cgroup and nvml data from the nvidia driver. It reports duration, cpu load, peak RAM, GPU load and peak GPU memory like so: |-- sprofile report (node03) -- Time: 0:00:03 / 1:00:00 CPU load: 2.0 / 4.0 RAM peak mem: 7G / 8G GPU load: 0.2 / 2.0 GPU peak mem: 7G / 40G| The requirements are to use the slurm cgroup plugin and to enable accounting on the GPU (nvidia-smi --accounting-mode=1). I hope you find this useful and let me know I you find bugs or want to contribute. Regards, Nicolas Granger

1 0

A fairshare policy that spans multiple clusters
by David Baker 05 Jan '24

05 Jan '24

Hello, We are soon to install new Slurm cluster at our site. That means that we will have a total of three clusters running Slurm. Only two, that is the new clusters, will share a common file system. The original cluster has its own file system is independent of the new arrivals. If possible, we would like to try to prevent users from making significant user of all the clusters and get a 'triple whammy'. In other words, is there any way to share the fairshare information between the clusters so that a user's use of one of the clusters impacts their usage on the other clusters – if that makes sense. Does anyone have any thoughts on this question, please? Am I correct in thinking that federating clusters is related to my question? Do I gather correctly, however, that federation only works if there is a common database on a shared file system? Best regards, David

2 1

slurmdb endpoints for slurmrestd
by Jackson, Gary L. 05 Jan '24

05 Jan '24

I get an HTTP 404 when I try to GET /slurmdb/v0.0.39/clusters or any other /slurmdb endpoint. I get this against multiple versions of Slurm, including 23.11.1. Using GET against /slurm/v0.0.39/ping works just fine. Is there something I need to do to turn slurmdb endpoints on? -- Gary

1 0

Slurp for sw builds
by Duane Ellis 04 Jan '24

04 Jan '24

In my case I would like to use a slurm cluster for a sw ci/cd like solution for building sw images My current scripted full system build takes 3-5 hours and is done serially we could easily find places where we can choose to build things in parallel hence the idea is to spawn parallel builds on the Linux slurm cluster example: we have a list of images we iterate over in a for loop to build each thing the steps are: cd somedir then type make or run a shell script in that directory The last step after the for loop would be wait for all of the child builds to complete Once all child jobs are done we have a single job that combines or packages all the intermediate images Really want to use slurm because our FPGA team will have a giant slurm linux cluster for Xilinix FPGA builds and those nodes can do what we need for sw purposes (reusing the existing cluster is a huge win for us) My question is this: Can somebody point me to some sw build examples for or using slurm? All I can seem to find is how to install I see the srun and sbatch command man pages but no good examples Bonus would be something that integrates into a gitlab runner example or Jenkins in some way All I can seem to find is how to install and administer slurm not a how to use slurm Sent from my iPhone

3 2

Fwd: Fairshare: users not added
by Alex Ninaber 04 Jan '24

04 Jan '24

Hi all, A problem on slurm-23.02.4-1, 10.6.16-MariaDB; Maria and Slurmctld in active/active, SlurmDB in active/off, shared IP. Shared spool via Gluster. DB is an upgraded version of Slurm from somewhere 2017 (upgraded various times). The question is whether we should give up and start from scratch or if there's an easy fix. Problem: whenever we add a new user and add it to sacctmgr, the user shows up properly in sacct/mgr – but never shows up with the sshare commands after running some jobs. After restarting slurm a couple of times it shows up. Problem seems to be there also in the previous version. Only error we can see in slurmdb log: [2023-12-21T09:43:30.586] error: slurm_persist_conn_open: Something happened with the receiving/processing of the persistent connection init message to 10.141.255.253:6817 : (null) [2023-12-21T09:43:30.586] error: slurmdb_send_accounting_update_persist: Unable to open connection to registered cluster cluster. [2023-12-21T09:43:30.586] error: slurm_receive_msg: No response to persist_init [2023-12-21T09:43:30.586] error: update cluster: No error to cluster at 10.141.255.253(6817) [2023-12-21T09:43:30.586] debug2: DBD_FINI: CLOSE:1 COMMIT:0 [2023-12-21T09:43:30.586] debug4: accounting_storage/as_mysql: acct_storage_p_commit: got 0 commits AccountingStorageType=accounting_storage/slurmdbd # jobaccounting JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/linux SlurmctldTimeout=60 SlurmdTimeout=60 TCPTimeout=60 MessageTimeout=60 Best regards, Alex

1 0

All nodes within one partition reboot unexpectedly
by Jinglei Hu 02 Jan '24

02 Jan '24

Hi all, I had a slurm partition gpu_gmx with the following configuration (Slurm version: 20.11.9): > NodeName=node[09-11] Gres=gpu:rtx4080:1 Sockets=1 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=62000 State=UNKNOWN > NodeName=node[12-14] Gres=gpu:rtx4070ti:1 Sockets=1 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=62000 State=UNKNOWN > PartitionName=gpu_gmx Nodes=node[09-14] Default=NO MaxTime=UNLIMITED State=UP A job running on node11 had a problem, which then triggered the reboot of all nodes (node[09-14]) within the same partition (cat /var/log/slurmctld.log): > [2023-12-26T23:04:23.200] Batch JobId=25061 missing from batch node node11 (not found BatchStartTime after startup), Requeuing job > [2023-12-26T23:04:23.200] _job_complete: JobId=25061 WTERMSIG 126 > [2023-12-26T23:04:23.200] _job_complete: JobId=25061 cancelled by node failure > [2023-12-26T23:04:23.200] _job_complete: requeue JobId=25061 due to node failure > [2023-12-26T23:04:23.200] _job_complete: JobId=25061 done > [2023-12-26T23:04:23.200] validate_node_specs: Node node11 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:23.222] validate_node_specs: Node node09 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:23.579] Batch JobId=25060 missing from batch node node10 (not found BatchStartTime after startup), Requeuing job > [2023-12-26T23:04:23.579] _job_complete: JobId=25060 WTERMSIG 126 > [2023-12-26T23:04:23.579] _job_complete: JobId=25060 cancelled by node failure > [2023-12-26T23:04:23.579] _job_complete: requeue JobId=25060 due to node failure > [2023-12-26T23:04:23.579] _job_complete: JobId=25060 done > [2023-12-26T23:04:23.579] validate_node_specs: Node node10 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:23.581] validate_node_specs: Node node14 unexpectedly rebooted boot_time=1703603051 last response=1703602983 > [2023-12-26T23:04:23.654] validate_node_specs: Node node13 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:24.681] validate_node_specs: Node node12 unexpectedly rebooted boot_time=1703603053 last response=1703602983 > [2023-12-27T04:46:42.461] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=25060 uid 0 > [2023-12-27T04:46:43.822] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=25061 uid 0 The operating systems are CentOS 7.9.2009 on master node, and CentOS 8.5.2111 on node[09-14]. Does anyone have a similar experience and have a clue how to resolve this? Thanks in advance. Best, Jinglei

1 0

2024

slurm-users ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users