- slurm-users - lists.schedmd.com

FAQ describing how to hold a job ignores scontrol subcommands specifically for that purpose
by urbanjost 25 Feb '24

25 Feb '24

There are scontrol subcommands uhold/hold/release/requeuehold that are ignored when describing how to place a job on hold in FAQ 21; and it is never explained why the method described therein is the best method, it just states it is. Does anyone know why the FAQ method is better than using the subcommands? Is it because the PRIORITY and/or NICE values are not altered (maybe)? The question is also about Running but the answer is just about Starting and not Suspending which is not quite as clear (I think "running" should be "starting" to make that clear; and/or how to suspend should be described as well). If the answer is not clear to anyone, I might turn this into a request for clarification in the Slurm bugzilla as a documentation change request but wanted to see if this was already clear to anyone and I am missing something. From FAQ: 21. How can I temporarily prevent a job from running (e.g. place it into a hold state)? The easiest way to do this is to change a job's earliest begin time (optionally set at job submit time using the --begin option). The example below places a job into hold state (preventing its initiation for 30 days) and later permitting it to start now. <METHOD I> $ scontrol update JobId=1234 StartTime=now+30days ... later ... $ scontrol update JobId=1234 StartTime=now Note: Empirically in METHOD I the JobId can be a <job_list> , which I initially thought required single JobIDs. No explanation is given on why METHOD I is best; and there are other methods that seem more intuitive. I wonder what is undesirable about the following method which I have been using -- using the scontrol(1) subcommands hold/uhold/release/requeuehold. <METHOD II> $ scontrol hold <job_list> # advantage to administrator as user cannot change $ scontrol uhold <job_list> $ scontrol release <job_list> Examples: $ scontrol uhold jobname=JOB_NAME $ scontrol uhold '[100-200],300,500' Using uhold the "Reason" changes to something easily identifying the job is being held, as "Reason=None" became "Reason=JobHeldUser which seems better that Method I in that regard. The downside might be PRIORITY changed to zero and then went to a very large value when released? Another method appears to be that setting PRIORITY to zero also places jobs in hold. <METHOD III> $ scontrol update jobid=373 Priority=0 $ scontrol release jobid=373 # sets to a very high value $ scontrol update jobid=373 Priority=11111 # put back to lower desired value Once lowered, does an optional setting prevent a user from raising PRIORITY(?) The manual says Only the Slurm administrator or root can increase job's priority. At least on my machine the "release" buts the priority to a very high value, and a regular user can lower the value back to the (probably) lower original value. I did not see it happening but there are some statements in the documentation that make me think not only PRIORITY but perhaps the NICE value might be changed by METHOD II and METHOD III, although I could not get the NICE value to be inadvertently changed. Sent with [Proton Mail](https://proton.me/) secure email.

1 1

slurm-23.11.3-1 with X11 and zram causing permission errors: error: _forkexec_slurmstepd: slurmstepd failed to send return code got 0: Resource temporarily unavailable; Requeue of JobId=530 returned an error: Only batch jobs are accepted or processed
by Robert Kudyba 24 Feb '24

24 Feb '24

We switched over from using systemctl for tmp.mount and change to zram, e.g., modprobe zram echo 20GB > /sys/block/zram0/disksize mkfs.xfs /dev/zram0 mount -o discard /dev/zram0 /tmp srun with --x11 was working before changing this. We're on RHEL 9. slurmctld logs show this whenever --x11 is used with srun: [2024-02-23T20:22:43.442] [529.extern] error: setup_x11_forward: failed to create temporary XAUTHORITY file: Permission denied [2024-02-23T20:22:43.442] [529.extern] error: x11 port forwarding setup failed [2024-02-23T20:22:43.442] error: _forkexec_slurmstepd: slurmstepd failed to send return code got 0: Resource temporarily unavailable [2024-02-23T20:22:43.443] Could not launch job 529 and not able to requeue it, cancelling job [2024-02-23T20:26:15.881] [530.extern] error: setup_x11_forward: failed to create temporary XAUTHORITY file: Permission denied [2024-02-23T20:26:15.881] [530.extern] error: x11 port forwarding setup failed [2024-02-23T20:26:15.882] error: _forkexec_slurmstepd: slurmstepd failed to send return code got 0: Resource temporarily unavailable [2024-02-23T20:26:15.883] Could not launch job 530 and not able to requeue it, cancelling job slurmd log entries from a node: [2024-02-23T20:26:15.859] sched: _slurm_rpc_allocate_resources JobId=530 NodeList=2402-node005 usec=1800 [2024-02-23T20:26:15.882] _slurm_rpc_requeue: Requeue of JobId=530 returned an error: Only batch jobs are accepted or processed [2024-02-23T20:26:15.883] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=530 uid 0 [2024-02-23T20:26:15.962] _slurm_rpc_complete_job_allocation: JobId=530 error Job/step already completing or completed srun -v --pty -t 0-4:00 --x11 --mem=10g srun: defined options srun: -------------------- -------------------- srun: account : me srun: mem : 10G srun: nodelist : our-node srun: pty : srun: time : 04:00:00 srun: verbose : 1 srun: x11 : all srun: -------------------- -------------------- srun: end of defined options srun: Waiting for resource configuration srun: error: Nodes our-node are still not ready srun: error: Something is wrong with the boot of the nodes. slurm.conf has PrologFlags=x11 set. /usr/bin/xauth is installed on each compute node. Is this a known issue with zram or is that just a red herring and there's something else wrong?

3 5

Slurm version 23.11.4 is now available
by Tim McMullan 22 Feb '24

22 Feb '24

We are pleased to announce the availability of Slurm version 23.11.4. The 23.11.4 release includes a number of fixes to stability and various bug fixes. Some notable changes include that VSZ is no longer being reported when using cgroup/v2 (this is not provided by the kernel), a warning has been added if using select/linear and tolology/tree together as this will not be supported in the next major release, and a backwards compatibility issue that caused jobs using --gpus to be rejected when submitted from 23.02 or 22.05. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.4 > ========================== > -- Fix a memory leak when updating partition nodes. > -- Don't leave a partition around if it fails to create with scontrol. > -- Fix segfault when creating partition with bad node list from scontrol. > -- Fix preserving partition nodes on bad node list update from scontrol. > -- Fix assertion in developer mode on a failed message unpack. > -- Fix repeat POWER_DOWN requests making the nodes available for ping. > -- Fix rebuilding job alias_list on restart when nodes are still powering up. > -- Fix INVALID nodes running health check. > -- Fix cloud/future nodes not setting addresses on invalid registration. > -- scrun - Remove the requirement to set the SCRUN_WORKING_DIR environment > variable. This was a regression in 23.11. > -- Add warning for using select/linear with topology/tree. > This combination will not be supported in the next major version. > -- Fix health check program not being run after first pass of all nodes when > using MaxNodeCount. > -- sacct - Set process exit code to one for all errors. > -- Add SlurmctldParameters=disable_triggers option. > -- Fix issue running steps when the allocation requested an exclusive > allocation shards along with shards. > -- Fix cleaning up the sleep process and the cgroup of the extern step if > slurm_spank_task_post_fork returns an error. > -- slurm_completion - Add missing --gres-flags= options > multiple-tasks-per-sharing and one-task-per-sharing. > -- scrun - Avoid race condition that could cause outbound network > communications to incorrectly rejected with an incomplete packet error. > -- scrun - Gracefully handle kernel giving invalid expected number of incoming > bytes for a connection causing incoming packet corruption resulting in > connection getting closed. > -- srun - return 1 when a step lauch fails > -- scrun - Avoid race condition that could cause deadlock during shutdown. > -- Fix scontrol listpids to work under dynamic node scenarios. > -- Add --tres-bind to --help and --usage output. > -- Add --gres-flags=allow-task-sharing to allow GPUs to still be accessible > among all tasks when binding GPUs to specific tasks. > -- Fix issue with CUDA_VISIBLE_DEVICES showing the same MIG device for all > tasks when using MIGs with --tres-per-task or --gpus-per-task. > -- slurmctld - Prevent a potential hang during shutdown/reconfigure if the > association cache thread was previously shut down. > -- scrun - Avoid race condition that could cause scrun to hang during > shutdown when connections have pending events. > -- scrun - Avoid excessive polling of connections during shutdown that could > needlessly cause 100% CPU usage on a thread. > -- sbcast - Use user identity from broadcast credential instead of looking it > up locally on the node. > -- scontrol - Remove "abort" option handling. > -- Fix an error message referring to the wrong RPC. > -- Fix memory leak on error when creating dynamic nodes. > -- Fix a slurmctld segfault when a cloud/dynamic node changes hostname on > registration. > -- Prevent a slurmctld deadlock if the gpu plugin fails to load when > creating a node. > -- Change a slurmctld fatal() to an error() when attempting to create a > dynamic node with a global autodetect set in gres.conf. > -- Fix leaving node records on error when creating nodes with scontrol. > -- scrun/sackd - Avoid race condition where shutdown could deadlock. > -- Fix a regression in 23.02.5 that caused pam_slurm_adopt to fail when > the user has multiple jobs on a node. > -- Add GLOB_SILENCE flag that silences the error message which will display if > an include directive attempts to use the "*" wildcard. > -- Fix jobs getting rejected when submitting with --gpus option from older > versions of job submission commands (23.02 and older). > -- cgroup/v2 - Return 0 for VSZ. Kernel cgroups do not provide this metric. > -- scrun - Avoid race condition where outbound RPCs could be corrupted. > -- scrun - Avoid race condition that could cause a crash while compiled in > debug mode. > -- gpu/rsmi - Disable gpu usage statistics when not using ROCM 6.0.0+ > -- Fix stuck processes and incorrect environment when using --get-user-env. > -- Avoid segfault in the slurmdbd when TrackWCKey=no but you are still using > use WCKeys. > -- Fix ctld segfault with TopologyParam=RoutePart and no partition defined. > -- slurmctld - Fix missing --deadline handling for jobs not evaluated by the > schedulers (i.e. non-runnable, skipped for other reasons, etc.). > -- Demote some eio related logs from error to verbose in user commands. These > are not generally actionable by the user and are easilly generated by port > scanning a machine running srun. > -- Make sprio correctly print array tasks that have not yet been split out. > -- topology/block - Restrict the number of last-level blocks in any allocation. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

RHEL 8.9+SLURM-23.11.3+MLNX_OFED_LINUX-23.10-1.1.9.0+ OpenMPI-5.0.2
by S L 20 Feb '24

20 Feb '24

Hi All, We're currently in the process of setting up SLURM on a RHEL 8.9 based cluster. Here's a summary of the steps we've taken so far: Installed MLNX OFED ConnectX-5.2. Compiled and installed PMiX and UCX. Compiled and installed Slurm with PMiX_v4 and UCX support. Compiled OpenMPI with SLURM, PMIx, libevent, and hwloc support. All compute nodes are reachable via the IB network. *Problem:* While hello world MPI jobs are working fine on multiple nodes, the jobs are not utilizing Infiniband. srun --mpi=pmix -N2 -n2 --ntasks-per-node=2 ./hello > log.out 2>&1 Output from srun --mpi=list: MPI plugin types are... none cray_shasta pmi2 pmix specific pmix plugin versions available: pmix_v4 Could someone please point me in the right direction on how to troubleshoot this issue? Thank you for your assistance. Sudhakar

2 1

URL for how to do for SLURM accounting setup
by John Joseph 20 Feb '24

20 Feb '24

Dear ALL, Good morning we were able to setup a test SLURM based system, with 4 nodes , Ubuntu 22.04 LTS and we were able to run COMSOL using "comsol batch" command Now we plan to have accounting https://slurm.schedmd.com/accounting.html Like to reach out and get guidance on any tutorial or how to do documentation on setting up accounting Appreciate your support Thanks Joseph John

2 2

CPU utilisation using two commands scontrol association and sreport makes a huge difference
by prachikakade.lit＠gmail.com 20 Feb '24

20 Feb '24

Dear Team, I created a small cluster of 3 nodes on my VM ware to work on the CPU utilization concept. I created a user name= hpcuser01, and allocated GrpTresMin=cpu=59000040 -> CPU minutes and gpu=0 Now, when I checked his utilization using scontrol association cmd # scontrol show ass user=hpcuser01 | grep -n15 hpcuser01 "" Output: QOS=hpcuser01_test(11) UsageRaw=5789263754.153907 GrpJobs=N(0) GrpJobsAccrue=N(5) GrpSubmitJobs=N(5) GrpWall=N(1188290.27) GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),billing=N(0),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0) GrpTRESMins=cpu=59000040(59001152),mem=N(293429427523),energy=N(0),node=N(1993778),billing=N(96073808),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=1(0) GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=46080(0),billing=N(0),fs/disk=N(0),vmem=N(0),pages=N(0)"" Here we can see that GrpTRESMins=cpu=59000040(59001152) after converting in Hours ( value /60) it will be: 983334(983352) CPU hours This means 983352 CPU hours have been used by hpcuser01 but, when I checked his utilization using sreport cmd #sreport -t hour --tres=cpu,gres/gpu,billing cluster AccountUtilizationByUser user=hpcuser01 start=2021-01-01T00:00:00 end=2024-02-20T23:59:59 -P | grep -iw -e billing -e cpu -e gres/gpu Output: cpu|979403 billing|1594902 gres/gpu|0 here I found that his utilisation is : 979403 I could like to understand why there is so much of a difference using two cmds to find out utilisation. Almost the difference is 979403 - 983352 = 3949 Request you to help me out with this. Regards, Prachi

1 0

Slurm Power Saving Guide: Why doesnt slurm mark as failed when resumeProgram returns =/= 0
by Xaver Stiensmeier 19 Feb '24

19 Feb '24

Dear slurm-user list, I had cases where our resumeProgram failed due to temporary cloud timeouts. In that case the resumeProgram returns a value =/= 0. Why does Slurm still wait until resumeTimeout instead of just accepting the startup as failed which then should lead to a rescheduling of the job. Is there some way to achieve the described effect i.e. tell Slurm: "You can stop waiting, the node won't come alive." or am I missing the correct way how this should be handled in Slurm? Best regards, Xaver

1 0

slurmdbd 17.02: "cluster not registered" (but things work)
by Matthias Leopold 19 Feb '24

19 Feb '24

Hi, I need to take care of a 17.02 Slurm cluster (I'm preparing it for upgrades). I see that slurmdbd logs various "cluster not registered" messages at startup (DBD_CLUSTER_TRES,DBD_JOB_START,DBD_STEP_START), but I don't see a real problem. Accounting works. Do I have to worry? Can this be related to upper/lower case issues with ClusterName? For some reason ClusterName is all upper case in slurm.conf. According to docs (and my experience with Slurm 21.08) though, this should be OK. Thx for help Matthias

1 0

Recover Batch Script Error
by Jason Simms 16 Feb '24

16 Feb '24

Hello all, I've used the "scontrol write batch_script" command to output the job submission script from completed jobs in the past, but for some reason, no matter which job I specify, it tells me it is invalid. Any way to troubleshoot this? Alternatively, is there another way - even if a manual database query - to recover the job script, assuming it exists in the database? sacct --jobs=38960 JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- 38960 amr_run_v+ tsmith2lab tsmith2lab 72 COMPLETED 0:0 38960.batch batch tsmith2lab 40 COMPLETED 0:0 38960.extern extern tsmith2lab 72 COMPLETED 0:0 38960.0 hydra_pmi+ tsmith2lab 72 COMPLETED 0:0 scontrol write batch_script 38960 job script retrieval failed: Invalid job id specified Warmest regards, Jason -- *Jason L. Simms, Ph.D., M.P.H.* Manager of Research Computing Swarthmore College Information Technology Services (610) 328-8102 Schedule a meeting: https://calendly.com/jlsimms

4 3

Need help managing licence
by Sylvain MARET 16 Feb '24

16 Feb '24

Hello everyone ! Recently our users bought a cplex dynamic license and want to use it on our slurm cluster. I've installed the paid version of cplex within modules so authorized user can load it with a simple module load cplex/2111 command but I don't know how to manage and ensure slurm doesn't launch a job if 20 people are already running code with this license. How do you guys manage paid licenses on your cluster ? Any advice would be appreciated ! Regards, Sylvain Maret

2 1

MPI/PMIx Issues after 23.11 Update
by Oliver Smith 16 Feb '24

16 Feb '24

Hi all, We’re running a small slurm dev cluster on Ubuntu and are facing issues with MPI/PMIx after upgrading slurm from 23.02.5 to 23.11.3. The first job step to use MPI within a job fails roughly 80% of the time but following attempts to use MPI within the same job work fine. For the failing job step we see this error after hitting the MPI timeout: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:741: 0x55a03f8d7a90: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: hpc-d-msh-01a02 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:759: 0x55a03f8d7a90: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:762: my peerid: 1:hpc-d-msh-01a02 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:769: neighbor id: next 0:hpc-d-msh-01a01, prev 0:hpc-d-msh-01a01 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:778: Context ptr=0x55a03f8d7b08, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:778: Context ptr=0x55a03f8d7b40, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:778: Context ptr=0x55a03f8d7b78, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:787: seq=0 contribs: loc=1/prev=1/fwd=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:791: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:824: done contrib: hpc-d-msh-01a01 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:826: wait contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:828: status=PMIXP_COLL_RING_FINILIZE slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:831: buf (offset/size): 0/33362 [hpc-d-msh-01a01.tds.hpc.barf1.com:47652] pml_ucx.c:178 Error: Failed to receive UCX worker address: Not found (-13) [hpc-d-msh-01a01.tds.hpc.barf1.com:47652] pml_ucx.c:477 Error: Failed to resolve UCX endpoint for rank 31 [hpc-d-msh-01a01:47652] *** An error occurred in MPI_Send [hpc-d-msh-01a01:47652] *** reported by process [683360612,0] [hpc-d-msh-01a01:47652] *** on communicator MPI_COMM_WORLD [hpc-d-msh-01a01:47652] *** MPI_ERR_OTHER: known error not in list [hpc-d-msh-01a01:47652] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [hpc-d-msh-01a01:47652] *** and potentially your MPI job) slurmstepd: error: *** STEP 1890.0 ON hpc-d-msh-01a01 CANCELLED AT 2024-02-14T16:14:52 *** OpenMPI/PMIx versions have not changed and downgrading slurm to 23.02.5 seems to resolve the issue. We’d appreciate any pointers anyone might have. Thanks Oli This email comprises confidential information of Mercedes-Benz Grand Prix Limited ("MGP") unless it contains an explicit statement to the contrary made by an authorised representative of MGP. Contracts may only be concluded on behalf of MGP by its authorised signatories and not solely by email communication. No employee, agent, contractor, consultant and/or other representative of MGP is authorised to conclude any legally binding agreement on behalf of MGP by email alone without the express prior written confirmation of two authorised signatories of MGP. Mercedes-Benz Grand Prix Limited. Registered in England No. 787446. Registered Office at Mercedes-Benz Grand Prix Limited, Operations Centre, Brackley, Northants NN13 7BD. Note: The MGP Legal Department also acts on behalf of Mercedes-Benz Motorsport Limited ("MBM") and the above notice applies mutatis mutandis in respect of all email communications of MBM. MBM: Mercedes-Benz Motorsport Limited. Registered in England No. 13057973. Registered office at Mercedes-Benz Motorsport Limited, Lauda Drive, Brackley, Northants NN13 7BD. Please consider the environment before printing this email.

1 0

slurmctld: slurm_bufs_sendto(msg_type=SRUN_STEP_SIGNAL) failed: Connection reset by peer
by Rike-Benjamin Schuppner 15 Feb '24

15 Feb '24

Hi, I am getting the following error in the logs whenever I run a few srun jobs in a batch. Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: debug: _send_timeout: Socket POLLERR: Connection reset by peer Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: error: slurm_send_node_msg: [socket:[921897]] slurm_bufs_sendto(msg_type=SRUN_STEP_SIGNAL) failed: Connection reset by peer Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: debug: laying out the 1 tasks on 1 hosts compute2 dist 1 The slurm version is 23.11.3 and an example sbatch file is: #!/bin/bash #SBATCH --job-name=slurm_test #SBATCH --mem=1gb #SBATCH --time=00:05:00 #SBATCH --output=slurm_test_%j.log pwd; hostname; date srun --exclusive -c1 -N1 -n1 bash -c "hostname ; sleep 10" & srun --exclusive -c1 -N1 -n1 bash -c "hostname ; sleep 10" & wait The more sruns I have in the script (&-backgrounded or not), the more the error shows up. Is there anything I could do to fix this error? Best /rike

2 1

Naive SLURM question: equivalent to LSF pre-exec
by Cutts, Tim 14 Feb '24

14 Feb '24

Hi, I apologise if I’ve failed to find this in the documentation (and am happy to be told to RTFM) but a recent issue for one of my users resulted in a question I couldn’t answer. LSF has a feature called a Pre-Exec where a script executes to check whether a node is ready to run a task. So, you can run arbitrary checks and go back to the queue if they fail. For example, if I have some automounted filesystems, and I want to be able to check for failure of the automounted, in an LSF world, I can do: bsub -E “test -f /nfs/someplace/file_I_know_exists” my_job.sh What’s the equivalent in SLURM? Thanks, Tim -- Tim Cutts Scientific Computing Platform Lead AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>

3 2

job_submit.lua - uid in Docker cluster
by Loris Bennett 14 Feb '24

14 Feb '24

Hi, Having used https://github.com/giovtorres/slurm-docker-cluster successfully a couple of years ago to develop a job_submit.lua plugin, I am trying to do this again. However, the plugin which works on our current cluster (CentOS 7.9, Slurm 23.02.7) fails in the Docker cluster (Rocky 8.9, Slurm 23.02.7) with error: job_submit/lua: /etc/slurm/job_submit.lua: /usr/share/lua/5.3/posix/deprecated.lua:453: bad argument #1 to 'getpwuid' (int expected, got number) which occurs when submit_user = posix.getpasswd(submit_user.uid) The problems is that 'submit_user.uid' has the value 0.0 and is thus not an integer. The only user within the Docker cluster is 'root'. Has anyone come across this issue? Is it to do with the Docker environment or the difference in the OS versions (Lua 5.1.4 vs. 5.3.4, lua-posix 32 vs. 33.3.1)? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT (ex-ZEDAT), Freie Universität Berlin

1 0

Increasing SlurmdTimeout beyond 300 Seconds
by Andrew Baughan (ITCS - Staff) 12 Feb '24

12 Feb '24

Hi, We've been experiencing issues with network saturation on our older nodes caused by storage (GPFS) backups. This causes slurmctld to loose contact with slurmd on some compute nodes and for user jobs to be killed. While the longer term solution is to replace these and upgrade the network, I'm wondering if there are any ramifications, beyond nodes with genuine issues taking longer to get marked down, by increasing SlurmdTimeout. We've already applied a modest increase which has helped but not resolved the issue and wondering if we should push it further in the interim. Kind Regards Andy Baughan HPC Systems Developer

4 3

simple question, I guess… from a newbie sysadmin
by Richard Randriatoamanana 12 Feb '24

12 Feb '24

Hi, I am trying to help a sysadmin colleague (and to understand for myself) trying to configure a new slurm server and he struggles to understand if there is an alternative way to config slurm managing job policy submission per user without necessarily installing an accounting mariadb service. I said no but maybe I misunderstand slurm admin configuration. Thanks in advance for feedback. Bests, Richard — Sent from my rotary and brandy phone ! Apologies for typos and any spelling errors

2 1

Question on using a suffix with "squeue --Format=FORMAT"
by urbanjost 10 Feb '24

10 Feb '24

I do not see how to use the suffix for the --Format option of the squeue(1) command. The description does not indicate how to tell the end of the size and period components and the beginning of the suffix, but assuming the suffix must being with something other than a period or numeric character I found all suffix values produce a null separator. A wanted to use --Format somewhat like --format=%all, producing all fields. As the --Format did not have an equivalent descriptor, (ie. something like --Format=All ) it looked like it would be required to list all the allowed keywords with a suffix of "|", which from the description I assumed would be as simple as a longer version of something like thi: squeue --Format='STDOUT:|,STDERR:|,STDIN:|,WorkDir:|' Given the description: -O, --Format=<output_format> Specify the information to be displayed. Also see the -o, --format=<output_format> option described above (which supports greater flexibility in formatting, but does not support access to all fields because we ran out of letters). Requests a comma separated list of job information to be displayed. The format of each field is "type[:[.][size][suffix]]" size Minimum field size. If no size is specified, 20 char‐ acters will be allocated to print the information. . Indicates the output should be right justified and size must be specified. By default output is left justified. suffix Arbitrary string to append to the end of the field. Note that many of these type specifications are valid only for jobs while others are valid only for job steps. Valid type specifications include: Even something like this produces no error but the suffix is always a null string: squeue --Format='STDOUT:SUFFIX,STDERR:SUFFIXSTDIN:SUFFIXWorkDir:SUFFIX So if I want to generate output with squeue --Format=FORMAT with a vertical line character between the output fields what is the proper syntax? A test bash(1) script where I could not get the SUFFIX variable to work as expected ... #!/bin/bash #@(#) Build value for squeue --Format='FORMAT' that includes all available fields with a vertical bar separating each field GOODNAMES='' BADNAMES='' SUFFIX='SUFFIX' GOODNAMES='' BADNAMES='' SUFFIX='1000:|' GOODNAMES='' BADNAMES='' SUFFIX='|' for NAME in STDOUT Account AccrueTime admin_comment AllocNodes AllocSID ArrayJobID ArrayTaskID AssocID BatchFlag BatchHost BoardsPerNode BurstBuffer BurstBufferState Cluster ClusterFeature Command Comment Container ContainerID Contiguous Cores CoreSpec CPUFreq cpus-per-task cpus-per-tres Deadline DelayBal Licenses MaxCPUs MaxNodes MCSLabel mem-per-tres MinCpus MinMemory MinTime Mintition PendingTime PreemptTime Prefer Priority PriorityLong Profile QOS Reason Riable SiblingsViableRaw Sockets SPerBoard StartTime State StateCompact STDERR STres-per-node tres-per-socket tres-per-step tres-per-task UserID UserName Wait4Sw do if squeue --Format="$NAME" >/dev/null then GOODNAMES="$GOODNAMES${NAME}:${SUFFIX}," else BADNAMES="$BADNAMES${NAME}," fi done 2>&1 #FORMAT=$(tr --squeeze-repeats ' ' ' '<<<$GOODNAMES) FORMAT=${GOODNAMES%%,} # trim off trailing comma read r c < <(stty size) # get screen width for use with cut(1) # try it squeue --Format="$FORMAT"|column -t|cut -c 1-$c # echo it echo "$FORMAT" exit Sent with [Proton Mail](https://proton.me/) secure email.

1 0

question on sbatch --prefer
by Alan Stange 10 Feb '24

10 Feb '24

Hello all, I'm somewhat new to Slurm, but long time user of other batch systems. Assume we have a simple cluster of uniform racks of systems with no special resources, and our jobs are all single cpu tasks. Lets say I have a long running job in the cluster, which needs to spawn a helper process into the cluster. We have a strong preference for this helper to run on the same cluster node as the original job, but if that node is already scheduled full, then we want this new task to be scheduled on another systems without any delay. The problem I have is that the --nodelist doesn't solve this, and, as far as I can tell, there's no option with --prefer to specify a node name as a resource, without creating a gres for every hostname in the cluster. It seems like what I'm trying to do should be achievable, but having read though the documentation and searched the archives of this list, I'm not seeing a solution. I'm hoping someone here has some experience with this and can point me in the right direction. Sincerely, Alan

3 3

Compilation question
by Sylvain MARET 10 Feb '24

10 Feb '24

Hello everyone ! I'm trying to compile slurm 22.05.11 on Rocky linux 8.7 with freeipmi support I've seen the documentation so I've done the configure step : ./configure --with-pmix=$PMIXHOME --with-ucx=$UCXHOME --with-nvml=$NVMLHOME --prefix=$SLURMHOME --with-freeipmi=/usr but when I run make I end up with the following error : /bin/sh ../../../../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -L/usr/lib64 -lhdf5_hl -lhdf5 -lsz -lz -ldl -lm -o sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -L../../../../../src/api/.libs -lslurmfull -ldl ../libhdf5_api.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o .libs/sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -Wl,--export-dynamic -L/usr/lib64 -L../../../../../src/api/.libs /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so ../.libs/libhdf5_api.a -lhdf5_hl -lhdf5 -lsz -lz -ldl -lpthread -lm -lresolv -pthread -Wl,-rpath -Wl,/softs/batch/slurm/22.05.11/lib/slurm sh5util.o:(.init_array+0x0): undefined reference to `__nv_init_env' sh5util.o:(.init_array+0x8): undefined reference to `__flushz' sh5util.o:(.init_array+0x10): undefined reference to `__daz' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_transfer_unique' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_sort_key_pairs' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrchr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_unsetenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_sort' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_for_each' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin__pgi_isnanld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_extra_conf_path' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_ctzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_running_in_slurmctld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mcopy1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_clzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_count' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin_va_gparg1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_destroy_config_key_pair' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xfree_ptr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_getenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_free_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_log_level' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mset8' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup_printf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_delete_first' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_append' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_error' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_init_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_log_var' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_parse_file' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_fatal' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_popcountll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrcmp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_destroy' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_setenvpf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_destroy' collect2: error: ld returned 1 exit status What can I do to resolve these undefined reference error ? Regards, Sylvain Maret

3 2

Re: Selecting only a subset of GPU's from all available GPU's
by Minulakshmi S 10 Feb '24

10 Feb '24

Hi Loris, I have different kind of GPU's in same node , and I beleive feature applies to particular node , and cant be applied to only a few GPU's connected to a single node. On Mon, 18 Dec 2023, 12:14 Loris Bennett, <loris.bennett(a)fu-berlin.de> wrote: > Hi Minu, > > Minulakshmi S <minulakshmi.s(a)gmail.com> writes: > > > I'm submitting jobs to a cluster via SLURM scheduler, and let's say I > have access to 8 GPUs in my cluster in same node. They are GPUs of type > A,B,C,D,E,F,G,H. I > > would like to submit a job that requests the use of GPUs of type A or B > or C but NOT of type D/E/F/G/H. So I need some type of OR logic with the > --gres flag. > > > > Eg : When I request GPU of type A , I can do sbatch –gres=gpu:TypeA:1, > I need to input a subset of GPU’s and let slurm schedule job utilizing one > of the GPU > > from this allowed list. > > > > Regards > > Minu > > Assuming the GPUs within a node are all of the same type, could you define > a feature for each GPU type, assign the features to the appropriate > nodes, and then run the job with > > --contraint=gpu_TypeA --gres=gpu:1 > > ? > > This is obviously rather clunky and it would be much nicer if > multiple GPUs types passed to '--gres' were ORed. > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin >

1 0

Memory used per node
by Gerhard Strangar 09 Feb '24

09 Feb '24

Hello, I'm wondering if there's a way to tell how much memory my job is using per node. I'm doing #SBATCH -n 256 srun solver inputfile When I run sacct -o maxvmsize, the result apparently is the maxmimum VSZ of the largest solver process, not the maximum of the sum of them all (unlike when calling mpirun instead). When I sstat -o TresUsageInMax, I get the memory summed up over all nodes being used. Can I get the maximum VSZ per node? Gerhard

2 1

Could not find group with gid even when they exist
by Nic Lewis 09 Feb '24

09 Feb '24

After upgrading to version 23.11.3 we started to get slammed with the following log messages from slurmctld "error: validate_group: Could not find group with gid <id>" This spans a handful of groups and repeats constantly, drowning out just about everything else. Attempting to do a lookup on the group shows that they exist on the scheduler node, same for all the submission and compute nodes. As far as I can tell, slurm should be able to locate the group in question. Jobs submitted from users within those groups go through just fine. They get scheduled, run, and clean up no problem. I'm at a loss on where to look next.

1 1

Relative QOS limits in sacctmgr?
by Chip Seraphine 09 Feb '24

09 Feb '24

Hello, TL,DR: How does the relative QOS flag work? I have a QOS and I want it to be collectively restricted to 50% of the reachable cores in the cluster. I’ve been managing this by dividing my core count to 2 to get N, and doing ‘sacctmgr update qos foobar set MaxTRES=cpu=N’. That’s fine, except that N changes frequently as worker nodes come and go, so I have an hourly cronjob that enforces this. The problem with *that* is the values you back out with ‘sacctmgr show qos’ are often in different units than you input (e.g. gigs instead of megs of ram), which means I need to do some annoying parsing, will miss minor rounding errors, etc. (Using JSON solves that problem but introduces others that I do not want to deal with.) So while looking for a better way I noticed that QOSs support a flag called “relative”, which says “the QOS limits will be treated as percentages of a cluster/partition instead of absolutes”. Perfect! However, it’s not very clear to me what that means in practice. For example, with the “relative” flag set, the MaxTRES units still seem to work the same, expect metric suffices (“G”, ”M”, etc) and won’t accept periods. Is anyone actually using this flag, and if so can you tell me what attributes it affects, and in what way? -- Chip Seraphine Grid Operations For support please use help-grid in email or slack. This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

1 0

Is there a way to list allocated/unallocated resources defined in a QoS?
by Alastair Neil 09 Feb '24

09 Feb '24

Slurm version 23.02.07 If I have a QoS defined that has a set number of say GPU devices set in the GrpTRES. Is there an easy way to generate a list of how much of the defined quota is allocated or conversely un-allocated? e.g.: Name|Priority|GraceTime|Preempt|PreemptExemptTime|PreemptMode|Flags|UsageThres|UsageFactor|GrpTRES|GrpTRESMins|GrpTRESRunMins|GrpJobs|GrpSubmit|GrpWall|MaxTRES|MaxTRESPerNode|MaxTRESMins|MaxWall|MaxTRESPU|MaxJobsPU|MaxSubmitPU|MaxTRESPA|MaxJobsPA|MaxSubmitPA|MinTRES| normal|0|00:00:00|||cluster|||1.000000|||||||||||cpu=3000,gres/gpu=20||||||| dept1|1|00:00:00|||cluster|||1.000000|cpu=256,gres/gpu:1g.10gb=16,gres/gpu:2g.20gb=8,gres/gpu:3g.40gb=8,gres/gpu:a100.80gb=8||||||||||||||||| dept2|1|00:00:00|||cluster|||1.000000|cpu=256,gres/gpu:1g.10gb=0,gres/gpu:2g.20gb=0,gres/gpu:3g.40gb=0,gres/gpu:a100.80gb=16||||||||||||||||| So dept1 and dept2 qos are set on the same partition. How can a user with access to one or other see if there are available resources in the partition?

2 1

Errors upgrading to 23.11.0
by Xaver Stiensmeier 08 Feb '24

08 Feb '24

Dear slurm-user list, I got this error: Unable to start service slurmctld: Job for slurmctld.service failed because the control process exited with error code.\nSee \"systemctl status slurmctld.service\" and \"journalctl -xeu slurmctld.service\" for details. but in slurmctld.service I see nothing suspicious: slurmctld.service - Slurm controller daemon Loaded: loaded (/lib/systemd/system/slurmctld.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/slurmctld.service.d └─override.conf Active: active (running) since Wed 2024-02-07 15:50:56 UTC; 19min ago Main PID: 51552 (slurmctld) Tasks: 21 (limit: 9363) Memory: 10.4M CPU: 1min 16.088s CGroup: /system.slice/slurmctld.service ├─51552 /usr/sbin/slurmctld --systemd └─51553 "slurmctld: slurmscriptd" "" "" "" "" "" "" Feb 07 15:58:21 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: sched: _slurm_rpc_allocate_resources JobId=3 NodeList=(null) usec=959 Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 WTERMSIG 2 Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 cancelled by interactive user Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 done Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _slurm_rpc_complete_job_allocation: JobId=3 error Job/step already completing or completed Feb 07 15:58:42 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: sched: _slurm_rpc_allocate_resources JobId=4 NodeList=cluster-master-2vt2bqh7ahec04c,cluster-worker-2vt2bqh7ahec04c-2 usec=512 Feb 07 16:06:04 cluster-master-2vt2bqh7ahec04c slurmctld[51553]: slurmctld: error: _run_script: JobId=0 resumeprog exit status 1:0 Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=4 WTERMSIG 2 Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=4 done Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _slurm_rpc_complete_job_allocation: JobId=4 error Job/step already completing or completed I am unsure how to debug this further. It might be coming from a previous problem I tried to fix (basically a few deprecated keys in the configuration). I will try to restart the entire cluster with the added changes to rule out any follow up errors, but maybe it's something obvious a fellow list user can see. Best regards, Xaver

2 2

2025

2024

slurm-users ----- 2025 ----- June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users