- slurm-users - lists.schedmd.com

canonical way to run longer shell/bash interactive job (instead of srun inside of screen/tmux at front-end)?
by Josef Dvoracek 28 Feb '24

28 Feb '24

What is the recommended way to run longer interactive job at your systems? Our how-to includes starting screen at front-end node and running srun with bash/zsh inside, but that indeed brings dependency between login node (with screen) and the compute node job. On systems with multiple front-ends users need to remember the login node where they have their screen session.. Are you anybody using something more advanced and still understandable by casual user of HPC? (I know Open On Demand, but often the use of native console has certain benefits. ) cheers josef

7 9

Enforcing relative resource restrictions in submission script
by Matthew R. Baney 28 Feb '24

28 Feb '24

Hello Slurm users, I'm trying to write a check in our job_submit.lua script that enforces relative resource requirements such as disallowing more than 4 CPUs or 48GB of memory per GPU. The QOS itself has a MaxTRESPerJob of cpu=32,gres/gpu=8,mem=384G (roughly one full node), but we're looking to prevent jobs from "stranding" GPUs, e.g., a 32 CPU/384GB memory job with only 1 GPU. I might be missing something obvious, but the rabbit hole I'm going down at the moment is trying to check all of the different ways job arguments could be set in the job descriptor. i.e., the following should all be disallowed: srun --gres=gpu:1 --mem=49G ... (tres_per_node, mem_per_node set in the descriptor) srun --gpus=1 --mem-per-gpu=49G ... (tres_per_job, mem_per_tres) srun --gres=gpu:1 --ntasks-per-gpu=5 ... (tres_per_node, num_tasks, ntasks_per_tres) srun --gpus=1 --ntasks=2 --mem-per-cpu=25G ... (tres_per_job, num_tasks, mem_per_cpu) ... Essentially what I'm looking for is a way to access the ReqTRES string from the job record before it exists, and then run some logic against that i.e., if (CPU count / GPU count) > 4 or (mem count / GPU count) > 48G, error out. Is something like this possible? Thanks, Matthew -- Matthew Baney Assistant Director of Computational Systems mbaney(a)umd.edu | (301) 405-6756 University of Maryland Institute for Advanced Computer Studies 3154 Brendan Iribe Center 8125 Paint Branch Dr. College Park, MD 20742

2 1

User-facing documentation on shard use
by wdennis＠nec-labs.com 28 Feb '24

28 Feb '24

Hello list, We have just enabled "gres/shard" in order to enable sharing of GPUs on our cluster. I am now looking for examples of user-facing documentation on this feature. If anyone has something, and can send a URL or other example, I'd appreciate it. Thanks, Will

1 0

Partition, Qos Limits & Scheduling of large jobs
by Muck, Katrin 28 Feb '24

28 Feb '24

Hi everyone! I read the slurm documentation about qos, resource limits, scheduling and priority now multiple times and even looked into the slurm source but I'm still not sure if I got everything correctly, so this is why I decided to ask here ... The problem: we see the effect that sometimes larger jobs with e.g. 16 gpus in our (small) gpu queue get delayed and shifted back without any reason that is apparent to us. small jobs that only use e.g. 1 or 2 gpus get scheduled much quicker even though they have a runtime of 3 days ... What we want to do: - We have a number of nodes with 2x gpus that are usable by the users of our cluster - Some of these nodes belong to so called 'private projects'. Private projects have higher priority than other projects. Attached to that is a contingent of nodes & "guaranteed" nodes e.g. they could have a contingent of 4 nodes (8 gpus) and e.g. 2 "guaranteed" nodes (4 gpus) - Guaranteed nodes are nodes that should always be kept idle for the private project, so users of the private project can immediately schedule work on those nodes - The other nodes are shared with other projects in general if they are not "in use" How we are currently doing this (it has history): Lets assume we have 50 nodes and 100 gpus. - We have a single partition for all gpu nodes (e.g. 50 nodes) - Private projects have private queues with a very high priority and a gres limit of the number of gpus they reserved (e.g. 10 nodes -> 20 gpus) - Normal projects only have access to the public queue and schedule work there. - This public queue has an upper gres limit of "total number of gpus" - "guaranteed gpus of all private projects" (e.g. 50 - 10 nodes -> 40 nodes -> 80 gpus). Regarding the scheduler, we currently use the following settings: SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_CORE_MEMORY,CR_CORE_DEFAULT_DIST_BLOCK,CR_ONE_TASK_PER_CORE SchedulerParameters=defer,max_sched_time=4,default_queue_depth=1000,partition_job_depth=500,enable_user_top,bf_max_job_user=20,bf_interval=120,bf_window=4320,bf_resolution=1800,bf_continue Partition/queue depth is deliberately set high at the moment to avoid problems with jobs not even being examined. The problem in more detail: One of the last jobs (16 gpus needed) we diagnosed had a approximate start time that was beyond all end times of running/scheduled jobs - like jobs on feb 22 would release more than enough gpus so the job could have been immediately scheduled afterwards, but the start time was still feb 23. Priority wise the job had the highest priority of all pending jobs for the partition. When we turned on scheduler debugging and increased log levels, we observed the following messages for this job: JobId=xxxx being held, if allowed the job request will exceed QOS xxxxx group max tres(gres/gpu) limit yy with already used yy + requested 16 followed by sched: JobId=2796696 delayed for accounting policy So to us this meant that the scheduler was always hitting the qos limits, which makes sense because the usage is always very high in the gpu queue and thus the job wasn't scheduled ... At first we were worried, that this meant that "held"/"delayed" jobs like this would never actually get scheduled when contention is high enough e.g. small jobs getting backfilled in and thus qos limits stay at max for a long time. But for some reason we could not determine the job eventually got scheduled at one point and then ran at the scheduled start time. Open Questions: - why it couldn't be scheduled in the first place. initially we thought (from the source code i looked into) the "delayed for accounting policy" prevents further scheduling in general, but since it was scheduled this assumption must be wrong? - why it was scheduled at some point. when it was scheduled, contention was still high and the qos limits definitely still applied - how we could modify the current setup so that the scheduling of larger jobs becomes "better" and more reproducible/explainable Apart from all of this I'm also asking myself if there is maybe a better way to setup a system that works the way we want? This got a bit long but I hope its clear enough :) Kind regards, Katrin

1 0

"Optimal" slurm configuration
by Max Grönke 26 Feb '24

26 Feb '24

Hello! In our current cluster the workflows are quite diverse (bunch of large, long (24-72h) jobs; medium size <4h job; and many small 1 node jobs). The current priority is fair share only (averaged on a ~few months timescale). For the new setup we would like to (1) discourage the 1 node jobs [especially the long ones] (2) have a fair use of the machine for the different cases. In particular, we found that the waiting time / runtime ratio is very large for the ~1-4h jobs. For this reason, I have been trying to read up on strategies in terms of partition / priority politics but could not find any good documentation in this regard (as https://slurm.schedmd.com/documentation.html is more focused on the technical aspects). Does anyone know any good references for this? In particular, we were thinking about (a) somehow have a long-term as well as shorter term fairshare count, i.e., lowering the priority if a user has something running already or (b) introduce a "small" partition for the <4h jobs with higher priority but we're unsure if this will block all the larger jobs to run... Any help or reading material is highly appreciated. Many thanks, Max

2 1

FAQ describing how to hold a job ignores scontrol subcommands specifically for that purpose
by urbanjost 25 Feb '24

25 Feb '24

There are scontrol subcommands uhold/hold/release/requeuehold that are ignored when describing how to place a job on hold in FAQ 21; and it is never explained why the method described therein is the best method, it just states it is. Does anyone know why the FAQ method is better than using the subcommands? Is it because the PRIORITY and/or NICE values are not altered (maybe)? The question is also about Running but the answer is just about Starting and not Suspending which is not quite as clear (I think "running" should be "starting" to make that clear; and/or how to suspend should be described as well). If the answer is not clear to anyone, I might turn this into a request for clarification in the Slurm bugzilla as a documentation change request but wanted to see if this was already clear to anyone and I am missing something. From FAQ: 21. How can I temporarily prevent a job from running (e.g. place it into a hold state)? The easiest way to do this is to change a job's earliest begin time (optionally set at job submit time using the --begin option). The example below places a job into hold state (preventing its initiation for 30 days) and later permitting it to start now. <METHOD I> $ scontrol update JobId=1234 StartTime=now+30days ... later ... $ scontrol update JobId=1234 StartTime=now Note: Empirically in METHOD I the JobId can be a <job_list> , which I initially thought required single JobIDs. No explanation is given on why METHOD I is best; and there are other methods that seem more intuitive. I wonder what is undesirable about the following method which I have been using -- using the scontrol(1) subcommands hold/uhold/release/requeuehold. <METHOD II> $ scontrol hold <job_list> # advantage to administrator as user cannot change $ scontrol uhold <job_list> $ scontrol release <job_list> Examples: $ scontrol uhold jobname=JOB_NAME $ scontrol uhold '[100-200],300,500' Using uhold the "Reason" changes to something easily identifying the job is being held, as "Reason=None" became "Reason=JobHeldUser which seems better that Method I in that regard. The downside might be PRIORITY changed to zero and then went to a very large value when released? Another method appears to be that setting PRIORITY to zero also places jobs in hold. <METHOD III> $ scontrol update jobid=373 Priority=0 $ scontrol release jobid=373 # sets to a very high value $ scontrol update jobid=373 Priority=11111 # put back to lower desired value Once lowered, does an optional setting prevent a user from raising PRIORITY(?) The manual says Only the Slurm administrator or root can increase job's priority. At least on my machine the "release" buts the priority to a very high value, and a regular user can lower the value back to the (probably) lower original value. I did not see it happening but there are some statements in the documentation that make me think not only PRIORITY but perhaps the NICE value might be changed by METHOD II and METHOD III, although I could not get the NICE value to be inadvertently changed. Sent with [Proton Mail](https://proton.me/) secure email.

1 1

slurm-23.11.3-1 with X11 and zram causing permission errors: error: _forkexec_slurmstepd: slurmstepd failed to send return code got 0: Resource temporarily unavailable; Requeue of JobId=530 returned an error: Only batch jobs are accepted or processed
by Robert Kudyba 24 Feb '24

24 Feb '24

We switched over from using systemctl for tmp.mount and change to zram, e.g., modprobe zram echo 20GB > /sys/block/zram0/disksize mkfs.xfs /dev/zram0 mount -o discard /dev/zram0 /tmp srun with --x11 was working before changing this. We're on RHEL 9. slurmctld logs show this whenever --x11 is used with srun: [2024-02-23T20:22:43.442] [529.extern] error: setup_x11_forward: failed to create temporary XAUTHORITY file: Permission denied [2024-02-23T20:22:43.442] [529.extern] error: x11 port forwarding setup failed [2024-02-23T20:22:43.442] error: _forkexec_slurmstepd: slurmstepd failed to send return code got 0: Resource temporarily unavailable [2024-02-23T20:22:43.443] Could not launch job 529 and not able to requeue it, cancelling job [2024-02-23T20:26:15.881] [530.extern] error: setup_x11_forward: failed to create temporary XAUTHORITY file: Permission denied [2024-02-23T20:26:15.881] [530.extern] error: x11 port forwarding setup failed [2024-02-23T20:26:15.882] error: _forkexec_slurmstepd: slurmstepd failed to send return code got 0: Resource temporarily unavailable [2024-02-23T20:26:15.883] Could not launch job 530 and not able to requeue it, cancelling job slurmd log entries from a node: [2024-02-23T20:26:15.859] sched: _slurm_rpc_allocate_resources JobId=530 NodeList=2402-node005 usec=1800 [2024-02-23T20:26:15.882] _slurm_rpc_requeue: Requeue of JobId=530 returned an error: Only batch jobs are accepted or processed [2024-02-23T20:26:15.883] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=530 uid 0 [2024-02-23T20:26:15.962] _slurm_rpc_complete_job_allocation: JobId=530 error Job/step already completing or completed srun -v --pty -t 0-4:00 --x11 --mem=10g srun: defined options srun: -------------------- -------------------- srun: account : me srun: mem : 10G srun: nodelist : our-node srun: pty : srun: time : 04:00:00 srun: verbose : 1 srun: x11 : all srun: -------------------- -------------------- srun: end of defined options srun: Waiting for resource configuration srun: error: Nodes our-node are still not ready srun: error: Something is wrong with the boot of the nodes. slurm.conf has PrologFlags=x11 set. /usr/bin/xauth is installed on each compute node. Is this a known issue with zram or is that just a red herring and there's something else wrong?

3 5

Slurm version 23.11.4 is now available
by Tim McMullan 22 Feb '24

22 Feb '24

We are pleased to announce the availability of Slurm version 23.11.4. The 23.11.4 release includes a number of fixes to stability and various bug fixes. Some notable changes include that VSZ is no longer being reported when using cgroup/v2 (this is not provided by the kernel), a warning has been added if using select/linear and tolology/tree together as this will not be supported in the next major release, and a backwards compatibility issue that caused jobs using --gpus to be rejected when submitted from 23.02 or 22.05. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.4 > ========================== > -- Fix a memory leak when updating partition nodes. > -- Don't leave a partition around if it fails to create with scontrol. > -- Fix segfault when creating partition with bad node list from scontrol. > -- Fix preserving partition nodes on bad node list update from scontrol. > -- Fix assertion in developer mode on a failed message unpack. > -- Fix repeat POWER_DOWN requests making the nodes available for ping. > -- Fix rebuilding job alias_list on restart when nodes are still powering up. > -- Fix INVALID nodes running health check. > -- Fix cloud/future nodes not setting addresses on invalid registration. > -- scrun - Remove the requirement to set the SCRUN_WORKING_DIR environment > variable. This was a regression in 23.11. > -- Add warning for using select/linear with topology/tree. > This combination will not be supported in the next major version. > -- Fix health check program not being run after first pass of all nodes when > using MaxNodeCount. > -- sacct - Set process exit code to one for all errors. > -- Add SlurmctldParameters=disable_triggers option. > -- Fix issue running steps when the allocation requested an exclusive > allocation shards along with shards. > -- Fix cleaning up the sleep process and the cgroup of the extern step if > slurm_spank_task_post_fork returns an error. > -- slurm_completion - Add missing --gres-flags= options > multiple-tasks-per-sharing and one-task-per-sharing. > -- scrun - Avoid race condition that could cause outbound network > communications to incorrectly rejected with an incomplete packet error. > -- scrun - Gracefully handle kernel giving invalid expected number of incoming > bytes for a connection causing incoming packet corruption resulting in > connection getting closed. > -- srun - return 1 when a step lauch fails > -- scrun - Avoid race condition that could cause deadlock during shutdown. > -- Fix scontrol listpids to work under dynamic node scenarios. > -- Add --tres-bind to --help and --usage output. > -- Add --gres-flags=allow-task-sharing to allow GPUs to still be accessible > among all tasks when binding GPUs to specific tasks. > -- Fix issue with CUDA_VISIBLE_DEVICES showing the same MIG device for all > tasks when using MIGs with --tres-per-task or --gpus-per-task. > -- slurmctld - Prevent a potential hang during shutdown/reconfigure if the > association cache thread was previously shut down. > -- scrun - Avoid race condition that could cause scrun to hang during > shutdown when connections have pending events. > -- scrun - Avoid excessive polling of connections during shutdown that could > needlessly cause 100% CPU usage on a thread. > -- sbcast - Use user identity from broadcast credential instead of looking it > up locally on the node. > -- scontrol - Remove "abort" option handling. > -- Fix an error message referring to the wrong RPC. > -- Fix memory leak on error when creating dynamic nodes. > -- Fix a slurmctld segfault when a cloud/dynamic node changes hostname on > registration. > -- Prevent a slurmctld deadlock if the gpu plugin fails to load when > creating a node. > -- Change a slurmctld fatal() to an error() when attempting to create a > dynamic node with a global autodetect set in gres.conf. > -- Fix leaving node records on error when creating nodes with scontrol. > -- scrun/sackd - Avoid race condition where shutdown could deadlock. > -- Fix a regression in 23.02.5 that caused pam_slurm_adopt to fail when > the user has multiple jobs on a node. > -- Add GLOB_SILENCE flag that silences the error message which will display if > an include directive attempts to use the "*" wildcard. > -- Fix jobs getting rejected when submitting with --gpus option from older > versions of job submission commands (23.02 and older). > -- cgroup/v2 - Return 0 for VSZ. Kernel cgroups do not provide this metric. > -- scrun - Avoid race condition where outbound RPCs could be corrupted. > -- scrun - Avoid race condition that could cause a crash while compiled in > debug mode. > -- gpu/rsmi - Disable gpu usage statistics when not using ROCM 6.0.0+ > -- Fix stuck processes and incorrect environment when using --get-user-env. > -- Avoid segfault in the slurmdbd when TrackWCKey=no but you are still using > use WCKeys. > -- Fix ctld segfault with TopologyParam=RoutePart and no partition defined. > -- slurmctld - Fix missing --deadline handling for jobs not evaluated by the > schedulers (i.e. non-runnable, skipped for other reasons, etc.). > -- Demote some eio related logs from error to verbose in user commands. These > are not generally actionable by the user and are easilly generated by port > scanning a machine running srun. > -- Make sprio correctly print array tasks that have not yet been split out. > -- topology/block - Restrict the number of last-level blocks in any allocation. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

RHEL 8.9+SLURM-23.11.3+MLNX_OFED_LINUX-23.10-1.1.9.0+ OpenMPI-5.0.2
by S L 20 Feb '24

20 Feb '24

Hi All, We're currently in the process of setting up SLURM on a RHEL 8.9 based cluster. Here's a summary of the steps we've taken so far: Installed MLNX OFED ConnectX-5.2. Compiled and installed PMiX and UCX. Compiled and installed Slurm with PMiX_v4 and UCX support. Compiled OpenMPI with SLURM, PMIx, libevent, and hwloc support. All compute nodes are reachable via the IB network. *Problem:* While hello world MPI jobs are working fine on multiple nodes, the jobs are not utilizing Infiniband. srun --mpi=pmix -N2 -n2 --ntasks-per-node=2 ./hello > log.out 2>&1 Output from srun --mpi=list: MPI plugin types are... none cray_shasta pmi2 pmix specific pmix plugin versions available: pmix_v4 Could someone please point me in the right direction on how to troubleshoot this issue? Thank you for your assistance. Sudhakar

2 1

URL for how to do for SLURM accounting setup
by John Joseph 20 Feb '24

20 Feb '24

Dear ALL, Good morning we were able to setup a test SLURM based system, with 4 nodes , Ubuntu 22.04 LTS and we were able to run COMSOL using "comsol batch" command Now we plan to have accounting https://slurm.schedmd.com/accounting.html Like to reach out and get guidance on any tutorial or how to do documentation on setting up accounting Appreciate your support Thanks Joseph John

2 2

CPU utilisation using two commands scontrol association and sreport makes a huge difference
by prachikakade.lit＠gmail.com 20 Feb '24

20 Feb '24

Dear Team, I created a small cluster of 3 nodes on my VM ware to work on the CPU utilization concept. I created a user name= hpcuser01, and allocated GrpTresMin=cpu=59000040 -> CPU minutes and gpu=0 Now, when I checked his utilization using scontrol association cmd # scontrol show ass user=hpcuser01 | grep -n15 hpcuser01 "" Output: QOS=hpcuser01_test(11) UsageRaw=5789263754.153907 GrpJobs=N(0) GrpJobsAccrue=N(5) GrpSubmitJobs=N(5) GrpWall=N(1188290.27) GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),billing=N(0),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0) GrpTRESMins=cpu=59000040(59001152),mem=N(293429427523),energy=N(0),node=N(1993778),billing=N(96073808),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=1(0) GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=46080(0),billing=N(0),fs/disk=N(0),vmem=N(0),pages=N(0)"" Here we can see that GrpTRESMins=cpu=59000040(59001152) after converting in Hours ( value /60) it will be: 983334(983352) CPU hours This means 983352 CPU hours have been used by hpcuser01 but, when I checked his utilization using sreport cmd #sreport -t hour --tres=cpu,gres/gpu,billing cluster AccountUtilizationByUser user=hpcuser01 start=2021-01-01T00:00:00 end=2024-02-20T23:59:59 -P | grep -iw -e billing -e cpu -e gres/gpu Output: cpu|979403 billing|1594902 gres/gpu|0 here I found that his utilisation is : 979403 I could like to understand why there is so much of a difference using two cmds to find out utilisation. Almost the difference is 979403 - 983352 = 3949 Request you to help me out with this. Regards, Prachi

1 0

Slurm Power Saving Guide: Why doesnt slurm mark as failed when resumeProgram returns =/= 0
by Xaver Stiensmeier 19 Feb '24

19 Feb '24

Dear slurm-user list, I had cases where our resumeProgram failed due to temporary cloud timeouts. In that case the resumeProgram returns a value =/= 0. Why does Slurm still wait until resumeTimeout instead of just accepting the startup as failed which then should lead to a rescheduling of the job. Is there some way to achieve the described effect i.e. tell Slurm: "You can stop waiting, the node won't come alive." or am I missing the correct way how this should be handled in Slurm? Best regards, Xaver

1 0

slurmdbd 17.02: "cluster not registered" (but things work)
by Matthias Leopold 19 Feb '24

19 Feb '24

Hi, I need to take care of a 17.02 Slurm cluster (I'm preparing it for upgrades). I see that slurmdbd logs various "cluster not registered" messages at startup (DBD_CLUSTER_TRES,DBD_JOB_START,DBD_STEP_START), but I don't see a real problem. Accounting works. Do I have to worry? Can this be related to upper/lower case issues with ClusterName? For some reason ClusterName is all upper case in slurm.conf. According to docs (and my experience with Slurm 21.08) though, this should be OK. Thx for help Matthias

1 0

Recover Batch Script Error
by Jason Simms 16 Feb '24

16 Feb '24

Hello all, I've used the "scontrol write batch_script" command to output the job submission script from completed jobs in the past, but for some reason, no matter which job I specify, it tells me it is invalid. Any way to troubleshoot this? Alternatively, is there another way - even if a manual database query - to recover the job script, assuming it exists in the database? sacct --jobs=38960 JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- 38960 amr_run_v+ tsmith2lab tsmith2lab 72 COMPLETED 0:0 38960.batch batch tsmith2lab 40 COMPLETED 0:0 38960.extern extern tsmith2lab 72 COMPLETED 0:0 38960.0 hydra_pmi+ tsmith2lab 72 COMPLETED 0:0 scontrol write batch_script 38960 job script retrieval failed: Invalid job id specified Warmest regards, Jason -- *Jason L. Simms, Ph.D., M.P.H.* Manager of Research Computing Swarthmore College Information Technology Services (610) 328-8102 Schedule a meeting: https://calendly.com/jlsimms

4 3

Need help managing licence
by Sylvain MARET 16 Feb '24

16 Feb '24

Hello everyone ! Recently our users bought a cplex dynamic license and want to use it on our slurm cluster. I've installed the paid version of cplex within modules so authorized user can load it with a simple module load cplex/2111 command but I don't know how to manage and ensure slurm doesn't launch a job if 20 people are already running code with this license. How do you guys manage paid licenses on your cluster ? Any advice would be appreciated ! Regards, Sylvain Maret

2 1

MPI/PMIx Issues after 23.11 Update
by Oliver Smith 16 Feb '24

16 Feb '24

Hi all, We’re running a small slurm dev cluster on Ubuntu and are facing issues with MPI/PMIx after upgrading slurm from 23.02.5 to 23.11.3. The first job step to use MPI within a job fails roughly 80% of the time but following attempts to use MPI within the same job work fine. For the failing job step we see this error after hitting the MPI timeout: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:741: 0x55a03f8d7a90: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: hpc-d-msh-01a02 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:759: 0x55a03f8d7a90: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:762: my peerid: 1:hpc-d-msh-01a02 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:769: neighbor id: next 0:hpc-d-msh-01a01, prev 0:hpc-d-msh-01a01 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:778: Context ptr=0x55a03f8d7b08, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:778: Context ptr=0x55a03f8d7b40, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:778: Context ptr=0x55a03f8d7b78, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:787: seq=0 contribs: loc=1/prev=1/fwd=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:791: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:824: done contrib: hpc-d-msh-01a01 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:826: wait contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:828: status=PMIXP_COLL_RING_FINILIZE slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:831: buf (offset/size): 0/33362 [hpc-d-msh-01a01.tds.hpc.barf1.com:47652] pml_ucx.c:178 Error: Failed to receive UCX worker address: Not found (-13) [hpc-d-msh-01a01.tds.hpc.barf1.com:47652] pml_ucx.c:477 Error: Failed to resolve UCX endpoint for rank 31 [hpc-d-msh-01a01:47652] *** An error occurred in MPI_Send [hpc-d-msh-01a01:47652] *** reported by process [683360612,0] [hpc-d-msh-01a01:47652] *** on communicator MPI_COMM_WORLD [hpc-d-msh-01a01:47652] *** MPI_ERR_OTHER: known error not in list [hpc-d-msh-01a01:47652] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [hpc-d-msh-01a01:47652] *** and potentially your MPI job) slurmstepd: error: *** STEP 1890.0 ON hpc-d-msh-01a01 CANCELLED AT 2024-02-14T16:14:52 *** OpenMPI/PMIx versions have not changed and downgrading slurm to 23.02.5 seems to resolve the issue. We’d appreciate any pointers anyone might have. Thanks Oli This email comprises confidential information of Mercedes-Benz Grand Prix Limited ("MGP") unless it contains an explicit statement to the contrary made by an authorised representative of MGP. Contracts may only be concluded on behalf of MGP by its authorised signatories and not solely by email communication. No employee, agent, contractor, consultant and/or other representative of MGP is authorised to conclude any legally binding agreement on behalf of MGP by email alone without the express prior written confirmation of two authorised signatories of MGP. Mercedes-Benz Grand Prix Limited. Registered in England No. 787446. Registered Office at Mercedes-Benz Grand Prix Limited, Operations Centre, Brackley, Northants NN13 7BD. Note: The MGP Legal Department also acts on behalf of Mercedes-Benz Motorsport Limited ("MBM") and the above notice applies mutatis mutandis in respect of all email communications of MBM. MBM: Mercedes-Benz Motorsport Limited. Registered in England No. 13057973. Registered office at Mercedes-Benz Motorsport Limited, Lauda Drive, Brackley, Northants NN13 7BD. Please consider the environment before printing this email.

1 0

slurmctld: slurm_bufs_sendto(msg_type=SRUN_STEP_SIGNAL) failed: Connection reset by peer
by Rike-Benjamin Schuppner 15 Feb '24

15 Feb '24

Hi, I am getting the following error in the logs whenever I run a few srun jobs in a batch. Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: debug: _send_timeout: Socket POLLERR: Connection reset by peer Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: error: slurm_send_node_msg: [socket:[921897]] slurm_bufs_sendto(msg_type=SRUN_STEP_SIGNAL) failed: Connection reset by peer Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: debug: laying out the 1 tasks on 1 hosts compute2 dist 1 The slurm version is 23.11.3 and an example sbatch file is: #!/bin/bash #SBATCH --job-name=slurm_test #SBATCH --mem=1gb #SBATCH --time=00:05:00 #SBATCH --output=slurm_test_%j.log pwd; hostname; date srun --exclusive -c1 -N1 -n1 bash -c "hostname ; sleep 10" & srun --exclusive -c1 -N1 -n1 bash -c "hostname ; sleep 10" & wait The more sruns I have in the script (&-backgrounded or not), the more the error shows up. Is there anything I could do to fix this error? Best /rike

2 1

Naive SLURM question: equivalent to LSF pre-exec
by Cutts, Tim 14 Feb '24

14 Feb '24

Hi, I apologise if I’ve failed to find this in the documentation (and am happy to be told to RTFM) but a recent issue for one of my users resulted in a question I couldn’t answer. LSF has a feature called a Pre-Exec where a script executes to check whether a node is ready to run a task. So, you can run arbitrary checks and go back to the queue if they fail. For example, if I have some automounted filesystems, and I want to be able to check for failure of the automounted, in an LSF world, I can do: bsub -E “test -f /nfs/someplace/file_I_know_exists” my_job.sh What’s the equivalent in SLURM? Thanks, Tim -- Tim Cutts Scientific Computing Platform Lead AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>

3 2

job_submit.lua - uid in Docker cluster
by Loris Bennett 14 Feb '24

14 Feb '24

Hi, Having used https://github.com/giovtorres/slurm-docker-cluster successfully a couple of years ago to develop a job_submit.lua plugin, I am trying to do this again. However, the plugin which works on our current cluster (CentOS 7.9, Slurm 23.02.7) fails in the Docker cluster (Rocky 8.9, Slurm 23.02.7) with error: job_submit/lua: /etc/slurm/job_submit.lua: /usr/share/lua/5.3/posix/deprecated.lua:453: bad argument #1 to 'getpwuid' (int expected, got number) which occurs when submit_user = posix.getpasswd(submit_user.uid) The problems is that 'submit_user.uid' has the value 0.0 and is thus not an integer. The only user within the Docker cluster is 'root'. Has anyone come across this issue? Is it to do with the Docker environment or the difference in the OS versions (Lua 5.1.4 vs. 5.3.4, lua-posix 32 vs. 33.3.1)? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT (ex-ZEDAT), Freie Universität Berlin

1 0

Increasing SlurmdTimeout beyond 300 Seconds
by Andrew Baughan (ITCS - Staff) 12 Feb '24

12 Feb '24

Hi, We've been experiencing issues with network saturation on our older nodes caused by storage (GPFS) backups. This causes slurmctld to loose contact with slurmd on some compute nodes and for user jobs to be killed. While the longer term solution is to replace these and upgrade the network, I'm wondering if there are any ramifications, beyond nodes with genuine issues taking longer to get marked down, by increasing SlurmdTimeout. We've already applied a modest increase which has helped but not resolved the issue and wondering if we should push it further in the interim. Kind Regards Andy Baughan HPC Systems Developer

4 3

simple question, I guess… from a newbie sysadmin
by Richard Randriatoamanana 12 Feb '24

12 Feb '24

Hi, I am trying to help a sysadmin colleague (and to understand for myself) trying to configure a new slurm server and he struggles to understand if there is an alternative way to config slurm managing job policy submission per user without necessarily installing an accounting mariadb service. I said no but maybe I misunderstand slurm admin configuration. Thanks in advance for feedback. Bests, Richard — Sent from my rotary and brandy phone ! Apologies for typos and any spelling errors

2 1

Question on using a suffix with "squeue --Format=FORMAT"
by urbanjost 10 Feb '24

10 Feb '24

I do not see how to use the suffix for the --Format option of the squeue(1) command. The description does not indicate how to tell the end of the size and period components and the beginning of the suffix, but assuming the suffix must being with something other than a period or numeric character I found all suffix values produce a null separator. A wanted to use --Format somewhat like --format=%all, producing all fields. As the --Format did not have an equivalent descriptor, (ie. something like --Format=All ) it looked like it would be required to list all the allowed keywords with a suffix of "|", which from the description I assumed would be as simple as a longer version of something like thi: squeue --Format='STDOUT:|,STDERR:|,STDIN:|,WorkDir:|' Given the description: -O, --Format=<output_format> Specify the information to be displayed. Also see the -o, --format=<output_format> option described above (which supports greater flexibility in formatting, but does not support access to all fields because we ran out of letters). Requests a comma separated list of job information to be displayed. The format of each field is "type[:[.][size][suffix]]" size Minimum field size. If no size is specified, 20 char‐ acters will be allocated to print the information. . Indicates the output should be right justified and size must be specified. By default output is left justified. suffix Arbitrary string to append to the end of the field. Note that many of these type specifications are valid only for jobs while others are valid only for job steps. Valid type specifications include: Even something like this produces no error but the suffix is always a null string: squeue --Format='STDOUT:SUFFIX,STDERR:SUFFIXSTDIN:SUFFIXWorkDir:SUFFIX So if I want to generate output with squeue --Format=FORMAT with a vertical line character between the output fields what is the proper syntax? A test bash(1) script where I could not get the SUFFIX variable to work as expected ... #!/bin/bash #@(#) Build value for squeue --Format='FORMAT' that includes all available fields with a vertical bar separating each field GOODNAMES='' BADNAMES='' SUFFIX='SUFFIX' GOODNAMES='' BADNAMES='' SUFFIX='1000:|' GOODNAMES='' BADNAMES='' SUFFIX='|' for NAME in STDOUT Account AccrueTime admin_comment AllocNodes AllocSID ArrayJobID ArrayTaskID AssocID BatchFlag BatchHost BoardsPerNode BurstBuffer BurstBufferState Cluster ClusterFeature Command Comment Container ContainerID Contiguous Cores CoreSpec CPUFreq cpus-per-task cpus-per-tres Deadline DelayBal Licenses MaxCPUs MaxNodes MCSLabel mem-per-tres MinCpus MinMemory MinTime Mintition PendingTime PreemptTime Prefer Priority PriorityLong Profile QOS Reason Riable SiblingsViableRaw Sockets SPerBoard StartTime State StateCompact STDERR STres-per-node tres-per-socket tres-per-step tres-per-task UserID UserName Wait4Sw do if squeue --Format="$NAME" >/dev/null then GOODNAMES="$GOODNAMES${NAME}:${SUFFIX}," else BADNAMES="$BADNAMES${NAME}," fi done 2>&1 #FORMAT=$(tr --squeeze-repeats ' ' ' '<<<$GOODNAMES) FORMAT=${GOODNAMES%%,} # trim off trailing comma read r c < <(stty size) # get screen width for use with cut(1) # try it squeue --Format="$FORMAT"|column -t|cut -c 1-$c # echo it echo "$FORMAT" exit Sent with [Proton Mail](https://proton.me/) secure email.

1 0

question on sbatch --prefer
by Alan Stange 10 Feb '24

10 Feb '24

Hello all, I'm somewhat new to Slurm, but long time user of other batch systems. Assume we have a simple cluster of uniform racks of systems with no special resources, and our jobs are all single cpu tasks. Lets say I have a long running job in the cluster, which needs to spawn a helper process into the cluster. We have a strong preference for this helper to run on the same cluster node as the original job, but if that node is already scheduled full, then we want this new task to be scheduled on another systems without any delay. The problem I have is that the --nodelist doesn't solve this, and, as far as I can tell, there's no option with --prefer to specify a node name as a resource, without creating a gres for every hostname in the cluster. It seems like what I'm trying to do should be achievable, but having read though the documentation and searched the archives of this list, I'm not seeing a solution. I'm hoping someone here has some experience with this and can point me in the right direction. Sincerely, Alan

3 3

Compilation question
by Sylvain MARET 10 Feb '24

10 Feb '24

Hello everyone ! I'm trying to compile slurm 22.05.11 on Rocky linux 8.7 with freeipmi support I've seen the documentation so I've done the configure step : ./configure --with-pmix=$PMIXHOME --with-ucx=$UCXHOME --with-nvml=$NVMLHOME --prefix=$SLURMHOME --with-freeipmi=/usr but when I run make I end up with the following error : /bin/sh ../../../../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -L/usr/lib64 -lhdf5_hl -lhdf5 -lsz -lz -ldl -lm -o sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -L../../../../../src/api/.libs -lslurmfull -ldl ../libhdf5_api.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o .libs/sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -Wl,--export-dynamic -L/usr/lib64 -L../../../../../src/api/.libs /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so ../.libs/libhdf5_api.a -lhdf5_hl -lhdf5 -lsz -lz -ldl -lpthread -lm -lresolv -pthread -Wl,-rpath -Wl,/softs/batch/slurm/22.05.11/lib/slurm sh5util.o:(.init_array+0x0): undefined reference to `__nv_init_env' sh5util.o:(.init_array+0x8): undefined reference to `__flushz' sh5util.o:(.init_array+0x10): undefined reference to `__daz' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_transfer_unique' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_sort_key_pairs' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrchr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_unsetenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_sort' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_for_each' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin__pgi_isnanld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_extra_conf_path' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_ctzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_running_in_slurmctld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mcopy1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_clzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_count' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin_va_gparg1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_destroy_config_key_pair' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xfree_ptr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_getenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_free_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_log_level' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mset8' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup_printf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_delete_first' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_append' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_error' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_init_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_log_var' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_parse_file' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_fatal' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_popcountll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrcmp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_destroy' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_setenvpf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_destroy' collect2: error: ld returned 1 exit status What can I do to resolve these undefined reference error ? Regards, Sylvain Maret

3 2

Re: Selecting only a subset of GPU's from all available GPU's
by Minulakshmi S 10 Feb '24

10 Feb '24

Hi Loris, I have different kind of GPU's in same node , and I beleive feature applies to particular node , and cant be applied to only a few GPU's connected to a single node. On Mon, 18 Dec 2023, 12:14 Loris Bennett, <loris.bennett(a)fu-berlin.de> wrote: > Hi Minu, > > Minulakshmi S <minulakshmi.s(a)gmail.com> writes: > > > I'm submitting jobs to a cluster via SLURM scheduler, and let's say I > have access to 8 GPUs in my cluster in same node. They are GPUs of type > A,B,C,D,E,F,G,H. I > > would like to submit a job that requests the use of GPUs of type A or B > or C but NOT of type D/E/F/G/H. So I need some type of OR logic with the > --gres flag. > > > > Eg : When I request GPU of type A , I can do sbatch –gres=gpu:TypeA:1, > I need to input a subset of GPU’s and let slurm schedule job utilizing one > of the GPU > > from this allowed list. > > > > Regards > > Minu > > Assuming the GPUs within a node are all of the same type, could you define > a feature for each GPU type, assign the features to the appropriate > nodes, and then run the job with > > --contraint=gpu_TypeA --gres=gpu:1 > > ? > > This is obviously rather clunky and it would be much nicer if > multiple GPUs types passed to '--gres' were ORed. > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin >

1 0