- slurm-users - lists.schedmd.com

Recover Batch Script Error
by Jason Simms 16 Feb '24

16 Feb '24

Hello all, I've used the "scontrol write batch_script" command to output the job submission script from completed jobs in the past, but for some reason, no matter which job I specify, it tells me it is invalid. Any way to troubleshoot this? Alternatively, is there another way - even if a manual database query - to recover the job script, assuming it exists in the database? sacct --jobs=38960 JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- 38960 amr_run_v+ tsmith2lab tsmith2lab 72 COMPLETED 0:0 38960.batch batch tsmith2lab 40 COMPLETED 0:0 38960.extern extern tsmith2lab 72 COMPLETED 0:0 38960.0 hydra_pmi+ tsmith2lab 72 COMPLETED 0:0 scontrol write batch_script 38960 job script retrieval failed: Invalid job id specified Warmest regards, Jason -- *Jason L. Simms, Ph.D., M.P.H.* Manager of Research Computing Swarthmore College Information Technology Services (610) 328-8102 Schedule a meeting: https://calendly.com/jlsimms

4 3

Need help managing licence
by Sylvain MARET 16 Feb '24

16 Feb '24

Hello everyone ! Recently our users bought a cplex dynamic license and want to use it on our slurm cluster. I've installed the paid version of cplex within modules so authorized user can load it with a simple module load cplex/2111 command but I don't know how to manage and ensure slurm doesn't launch a job if 20 people are already running code with this license. How do you guys manage paid licenses on your cluster ? Any advice would be appreciated ! Regards, Sylvain Maret

2 1

MPI/PMIx Issues after 23.11 Update
by Oliver Smith 16 Feb '24

16 Feb '24

Hi all, We’re running a small slurm dev cluster on Ubuntu and are facing issues with MPI/PMIx after upgrading slurm from 23.02.5 to 23.11.3. The first job step to use MPI within a job fails roughly 80% of the time but following attempts to use MPI within the same job work fine. For the failing job step we see this error after hitting the MPI timeout: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:741: 0x55a03f8d7a90: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: hpc-d-msh-01a02 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:759: 0x55a03f8d7a90: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:762: my peerid: 1:hpc-d-msh-01a02 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:769: neighbor id: next 0:hpc-d-msh-01a01, prev 0:hpc-d-msh-01a01 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:778: Context ptr=0x55a03f8d7b08, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:778: Context ptr=0x55a03f8d7b40, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:778: Context ptr=0x55a03f8d7b78, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:787: seq=0 contribs: loc=1/prev=1/fwd=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:791: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:824: done contrib: hpc-d-msh-01a01 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:826: wait contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:828: status=PMIXP_COLL_RING_FINILIZE slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: hpc-d-msh-01a02 [1]: pmixp_coll_ring.c:831: buf (offset/size): 0/33362 [hpc-d-msh-01a01.tds.hpc.barf1.com:47652] pml_ucx.c:178 Error: Failed to receive UCX worker address: Not found (-13) [hpc-d-msh-01a01.tds.hpc.barf1.com:47652] pml_ucx.c:477 Error: Failed to resolve UCX endpoint for rank 31 [hpc-d-msh-01a01:47652] *** An error occurred in MPI_Send [hpc-d-msh-01a01:47652] *** reported by process [683360612,0] [hpc-d-msh-01a01:47652] *** on communicator MPI_COMM_WORLD [hpc-d-msh-01a01:47652] *** MPI_ERR_OTHER: known error not in list [hpc-d-msh-01a01:47652] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [hpc-d-msh-01a01:47652] *** and potentially your MPI job) slurmstepd: error: *** STEP 1890.0 ON hpc-d-msh-01a01 CANCELLED AT 2024-02-14T16:14:52 *** OpenMPI/PMIx versions have not changed and downgrading slurm to 23.02.5 seems to resolve the issue. We’d appreciate any pointers anyone might have. Thanks Oli This email comprises confidential information of Mercedes-Benz Grand Prix Limited ("MGP") unless it contains an explicit statement to the contrary made by an authorised representative of MGP. Contracts may only be concluded on behalf of MGP by its authorised signatories and not solely by email communication. No employee, agent, contractor, consultant and/or other representative of MGP is authorised to conclude any legally binding agreement on behalf of MGP by email alone without the express prior written confirmation of two authorised signatories of MGP. Mercedes-Benz Grand Prix Limited. Registered in England No. 787446. Registered Office at Mercedes-Benz Grand Prix Limited, Operations Centre, Brackley, Northants NN13 7BD. Note: The MGP Legal Department also acts on behalf of Mercedes-Benz Motorsport Limited ("MBM") and the above notice applies mutatis mutandis in respect of all email communications of MBM. MBM: Mercedes-Benz Motorsport Limited. Registered in England No. 13057973. Registered office at Mercedes-Benz Motorsport Limited, Lauda Drive, Brackley, Northants NN13 7BD. Please consider the environment before printing this email.

1 0

slurmctld: slurm_bufs_sendto(msg_type=SRUN_STEP_SIGNAL) failed: Connection reset by peer
by Rike-Benjamin Schuppner 15 Feb '24

15 Feb '24

Hi, I am getting the following error in the logs whenever I run a few srun jobs in a batch. Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: debug: _send_timeout: Socket POLLERR: Connection reset by peer Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: error: slurm_send_node_msg: [socket:[921897]] slurm_bufs_sendto(msg_type=SRUN_STEP_SIGNAL) failed: Connection reset by peer Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: debug: laying out the 1 tasks on 1 hosts compute2 dist 1 The slurm version is 23.11.3 and an example sbatch file is: #!/bin/bash #SBATCH --job-name=slurm_test #SBATCH --mem=1gb #SBATCH --time=00:05:00 #SBATCH --output=slurm_test_%j.log pwd; hostname; date srun --exclusive -c1 -N1 -n1 bash -c "hostname ; sleep 10" & srun --exclusive -c1 -N1 -n1 bash -c "hostname ; sleep 10" & wait The more sruns I have in the script (&-backgrounded or not), the more the error shows up. Is there anything I could do to fix this error? Best /rike

2 1

Naive SLURM question: equivalent to LSF pre-exec
by Cutts, Tim 14 Feb '24

14 Feb '24

Hi, I apologise if I’ve failed to find this in the documentation (and am happy to be told to RTFM) but a recent issue for one of my users resulted in a question I couldn’t answer. LSF has a feature called a Pre-Exec where a script executes to check whether a node is ready to run a task. So, you can run arbitrary checks and go back to the queue if they fail. For example, if I have some automounted filesystems, and I want to be able to check for failure of the automounted, in an LSF world, I can do: bsub -E “test -f /nfs/someplace/file_I_know_exists” my_job.sh What’s the equivalent in SLURM? Thanks, Tim -- Tim Cutts Scientific Computing Platform Lead AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>

3 2

job_submit.lua - uid in Docker cluster
by Loris Bennett 14 Feb '24

14 Feb '24

Hi, Having used https://github.com/giovtorres/slurm-docker-cluster successfully a couple of years ago to develop a job_submit.lua plugin, I am trying to do this again. However, the plugin which works on our current cluster (CentOS 7.9, Slurm 23.02.7) fails in the Docker cluster (Rocky 8.9, Slurm 23.02.7) with error: job_submit/lua: /etc/slurm/job_submit.lua: /usr/share/lua/5.3/posix/deprecated.lua:453: bad argument #1 to 'getpwuid' (int expected, got number) which occurs when submit_user = posix.getpasswd(submit_user.uid) The problems is that 'submit_user.uid' has the value 0.0 and is thus not an integer. The only user within the Docker cluster is 'root'. Has anyone come across this issue? Is it to do with the Docker environment or the difference in the OS versions (Lua 5.1.4 vs. 5.3.4, lua-posix 32 vs. 33.3.1)? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT (ex-ZEDAT), Freie Universität Berlin

1 0

Increasing SlurmdTimeout beyond 300 Seconds
by Andrew Baughan (ITCS - Staff) 12 Feb '24

12 Feb '24

Hi, We've been experiencing issues with network saturation on our older nodes caused by storage (GPFS) backups. This causes slurmctld to loose contact with slurmd on some compute nodes and for user jobs to be killed. While the longer term solution is to replace these and upgrade the network, I'm wondering if there are any ramifications, beyond nodes with genuine issues taking longer to get marked down, by increasing SlurmdTimeout. We've already applied a modest increase which has helped but not resolved the issue and wondering if we should push it further in the interim. Kind Regards Andy Baughan HPC Systems Developer

4 3

simple question, I guess… from a newbie sysadmin
by Richard Randriatoamanana 12 Feb '24

12 Feb '24

Hi, I am trying to help a sysadmin colleague (and to understand for myself) trying to configure a new slurm server and he struggles to understand if there is an alternative way to config slurm managing job policy submission per user without necessarily installing an accounting mariadb service. I said no but maybe I misunderstand slurm admin configuration. Thanks in advance for feedback. Bests, Richard — Sent from my rotary and brandy phone ! Apologies for typos and any spelling errors

2 1

Question on using a suffix with "squeue --Format=FORMAT"
by urbanjost 10 Feb '24

10 Feb '24

I do not see how to use the suffix for the --Format option of the squeue(1) command. The description does not indicate how to tell the end of the size and period components and the beginning of the suffix, but assuming the suffix must being with something other than a period or numeric character I found all suffix values produce a null separator. A wanted to use --Format somewhat like --format=%all, producing all fields. As the --Format did not have an equivalent descriptor, (ie. something like --Format=All ) it looked like it would be required to list all the allowed keywords with a suffix of "|", which from the description I assumed would be as simple as a longer version of something like thi: squeue --Format='STDOUT:|,STDERR:|,STDIN:|,WorkDir:|' Given the description: -O, --Format=<output_format> Specify the information to be displayed. Also see the -o, --format=<output_format> option described above (which supports greater flexibility in formatting, but does not support access to all fields because we ran out of letters). Requests a comma separated list of job information to be displayed. The format of each field is "type[:[.][size][suffix]]" size Minimum field size. If no size is specified, 20 char‐ acters will be allocated to print the information. . Indicates the output should be right justified and size must be specified. By default output is left justified. suffix Arbitrary string to append to the end of the field. Note that many of these type specifications are valid only for jobs while others are valid only for job steps. Valid type specifications include: Even something like this produces no error but the suffix is always a null string: squeue --Format='STDOUT:SUFFIX,STDERR:SUFFIXSTDIN:SUFFIXWorkDir:SUFFIX So if I want to generate output with squeue --Format=FORMAT with a vertical line character between the output fields what is the proper syntax? A test bash(1) script where I could not get the SUFFIX variable to work as expected ... #!/bin/bash #@(#) Build value for squeue --Format='FORMAT' that includes all available fields with a vertical bar separating each field GOODNAMES='' BADNAMES='' SUFFIX='SUFFIX' GOODNAMES='' BADNAMES='' SUFFIX='1000:|' GOODNAMES='' BADNAMES='' SUFFIX='|' for NAME in STDOUT Account AccrueTime admin_comment AllocNodes AllocSID ArrayJobID ArrayTaskID AssocID BatchFlag BatchHost BoardsPerNode BurstBuffer BurstBufferState Cluster ClusterFeature Command Comment Container ContainerID Contiguous Cores CoreSpec CPUFreq cpus-per-task cpus-per-tres Deadline DelayBal Licenses MaxCPUs MaxNodes MCSLabel mem-per-tres MinCpus MinMemory MinTime Mintition PendingTime PreemptTime Prefer Priority PriorityLong Profile QOS Reason Riable SiblingsViableRaw Sockets SPerBoard StartTime State StateCompact STDERR STres-per-node tres-per-socket tres-per-step tres-per-task UserID UserName Wait4Sw do if squeue --Format="$NAME" >/dev/null then GOODNAMES="$GOODNAMES${NAME}:${SUFFIX}," else BADNAMES="$BADNAMES${NAME}," fi done 2>&1 #FORMAT=$(tr --squeeze-repeats ' ' ' '<<<$GOODNAMES) FORMAT=${GOODNAMES%%,} # trim off trailing comma read r c < <(stty size) # get screen width for use with cut(1) # try it squeue --Format="$FORMAT"|column -t|cut -c 1-$c # echo it echo "$FORMAT" exit Sent with [Proton Mail](https://proton.me/) secure email.

1 0

question on sbatch --prefer
by Alan Stange 10 Feb '24

10 Feb '24

Hello all, I'm somewhat new to Slurm, but long time user of other batch systems. Assume we have a simple cluster of uniform racks of systems with no special resources, and our jobs are all single cpu tasks. Lets say I have a long running job in the cluster, which needs to spawn a helper process into the cluster. We have a strong preference for this helper to run on the same cluster node as the original job, but if that node is already scheduled full, then we want this new task to be scheduled on another systems without any delay. The problem I have is that the --nodelist doesn't solve this, and, as far as I can tell, there's no option with --prefer to specify a node name as a resource, without creating a gres for every hostname in the cluster. It seems like what I'm trying to do should be achievable, but having read though the documentation and searched the archives of this list, I'm not seeing a solution. I'm hoping someone here has some experience with this and can point me in the right direction. Sincerely, Alan

3 3

Compilation question
by Sylvain MARET 10 Feb '24

10 Feb '24

Hello everyone ! I'm trying to compile slurm 22.05.11 on Rocky linux 8.7 with freeipmi support I've seen the documentation so I've done the configure step : ./configure --with-pmix=$PMIXHOME --with-ucx=$UCXHOME --with-nvml=$NVMLHOME --prefix=$SLURMHOME --with-freeipmi=/usr but when I run make I end up with the following error : /bin/sh ../../../../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -L/usr/lib64 -lhdf5_hl -lhdf5 -lsz -lz -ldl -lm -o sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -L../../../../../src/api/.libs -lslurmfull -ldl ../libhdf5_api.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o .libs/sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -Wl,--export-dynamic -L/usr/lib64 -L../../../../../src/api/.libs /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so ../.libs/libhdf5_api.a -lhdf5_hl -lhdf5 -lsz -lz -ldl -lpthread -lm -lresolv -pthread -Wl,-rpath -Wl,/softs/batch/slurm/22.05.11/lib/slurm sh5util.o:(.init_array+0x0): undefined reference to `__nv_init_env' sh5util.o:(.init_array+0x8): undefined reference to `__flushz' sh5util.o:(.init_array+0x10): undefined reference to `__daz' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_transfer_unique' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_sort_key_pairs' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrchr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_unsetenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_sort' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_for_each' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin__pgi_isnanld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_extra_conf_path' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_ctzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_running_in_slurmctld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mcopy1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_clzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_count' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin_va_gparg1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_destroy_config_key_pair' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xfree_ptr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_getenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_free_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_log_level' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mset8' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup_printf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_delete_first' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_append' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_error' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_init_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_log_var' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_parse_file' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_fatal' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_popcountll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrcmp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_destroy' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_setenvpf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_destroy' collect2: error: ld returned 1 exit status What can I do to resolve these undefined reference error ? Regards, Sylvain Maret

3 2

Re: Selecting only a subset of GPU's from all available GPU's
by Minulakshmi S 10 Feb '24

10 Feb '24

Hi Loris, I have different kind of GPU's in same node , and I beleive feature applies to particular node , and cant be applied to only a few GPU's connected to a single node. On Mon, 18 Dec 2023, 12:14 Loris Bennett, <loris.bennett(a)fu-berlin.de> wrote: > Hi Minu, > > Minulakshmi S <minulakshmi.s(a)gmail.com> writes: > > > I'm submitting jobs to a cluster via SLURM scheduler, and let's say I > have access to 8 GPUs in my cluster in same node. They are GPUs of type > A,B,C,D,E,F,G,H. I > > would like to submit a job that requests the use of GPUs of type A or B > or C but NOT of type D/E/F/G/H. So I need some type of OR logic with the > --gres flag. > > > > Eg : When I request GPU of type A , I can do sbatch –gres=gpu:TypeA:1, > I need to input a subset of GPU’s and let slurm schedule job utilizing one > of the GPU > > from this allowed list. > > > > Regards > > Minu > > Assuming the GPUs within a node are all of the same type, could you define > a feature for each GPU type, assign the features to the appropriate > nodes, and then run the job with > > --contraint=gpu_TypeA --gres=gpu:1 > > ? > > This is obviously rather clunky and it would be much nicer if > multiple GPUs types passed to '--gres' were ORed. > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin >

1 0

Memory used per node
by Gerhard Strangar 09 Feb '24

09 Feb '24

Hello, I'm wondering if there's a way to tell how much memory my job is using per node. I'm doing #SBATCH -n 256 srun solver inputfile When I run sacct -o maxvmsize, the result apparently is the maxmimum VSZ of the largest solver process, not the maximum of the sum of them all (unlike when calling mpirun instead). When I sstat -o TresUsageInMax, I get the memory summed up over all nodes being used. Can I get the maximum VSZ per node? Gerhard

2 1

Could not find group with gid even when they exist
by Nic Lewis 09 Feb '24

09 Feb '24

After upgrading to version 23.11.3 we started to get slammed with the following log messages from slurmctld "error: validate_group: Could not find group with gid <id>" This spans a handful of groups and repeats constantly, drowning out just about everything else. Attempting to do a lookup on the group shows that they exist on the scheduler node, same for all the submission and compute nodes. As far as I can tell, slurm should be able to locate the group in question. Jobs submitted from users within those groups go through just fine. They get scheduled, run, and clean up no problem. I'm at a loss on where to look next.

1 1

Relative QOS limits in sacctmgr?
by Chip Seraphine 09 Feb '24

09 Feb '24

Hello, TL,DR: How does the relative QOS flag work? I have a QOS and I want it to be collectively restricted to 50% of the reachable cores in the cluster. I’ve been managing this by dividing my core count to 2 to get N, and doing ‘sacctmgr update qos foobar set MaxTRES=cpu=N’. That’s fine, except that N changes frequently as worker nodes come and go, so I have an hourly cronjob that enforces this. The problem with *that* is the values you back out with ‘sacctmgr show qos’ are often in different units than you input (e.g. gigs instead of megs of ram), which means I need to do some annoying parsing, will miss minor rounding errors, etc. (Using JSON solves that problem but introduces others that I do not want to deal with.) So while looking for a better way I noticed that QOSs support a flag called “relative”, which says “the QOS limits will be treated as percentages of a cluster/partition instead of absolutes”. Perfect! However, it’s not very clear to me what that means in practice. For example, with the “relative” flag set, the MaxTRES units still seem to work the same, expect metric suffices (“G”, ”M”, etc) and won’t accept periods. Is anyone actually using this flag, and if so can you tell me what attributes it affects, and in what way? -- Chip Seraphine Grid Operations For support please use help-grid in email or slack. This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

1 0

Is there a way to list allocated/unallocated resources defined in a QoS?
by Alastair Neil 09 Feb '24

09 Feb '24

Slurm version 23.02.07 If I have a QoS defined that has a set number of say GPU devices set in the GrpTRES. Is there an easy way to generate a list of how much of the defined quota is allocated or conversely un-allocated? e.g.: Name|Priority|GraceTime|Preempt|PreemptExemptTime|PreemptMode|Flags|UsageThres|UsageFactor|GrpTRES|GrpTRESMins|GrpTRESRunMins|GrpJobs|GrpSubmit|GrpWall|MaxTRES|MaxTRESPerNode|MaxTRESMins|MaxWall|MaxTRESPU|MaxJobsPU|MaxSubmitPU|MaxTRESPA|MaxJobsPA|MaxSubmitPA|MinTRES| normal|0|00:00:00|||cluster|||1.000000|||||||||||cpu=3000,gres/gpu=20||||||| dept1|1|00:00:00|||cluster|||1.000000|cpu=256,gres/gpu:1g.10gb=16,gres/gpu:2g.20gb=8,gres/gpu:3g.40gb=8,gres/gpu:a100.80gb=8||||||||||||||||| dept2|1|00:00:00|||cluster|||1.000000|cpu=256,gres/gpu:1g.10gb=0,gres/gpu:2g.20gb=0,gres/gpu:3g.40gb=0,gres/gpu:a100.80gb=16||||||||||||||||| So dept1 and dept2 qos are set on the same partition. How can a user with access to one or other see if there are available resources in the partition?

2 1

Errors upgrading to 23.11.0
by Xaver Stiensmeier 08 Feb '24

08 Feb '24

Dear slurm-user list, I got this error: Unable to start service slurmctld: Job for slurmctld.service failed because the control process exited with error code.\nSee \"systemctl status slurmctld.service\" and \"journalctl -xeu slurmctld.service\" for details. but in slurmctld.service I see nothing suspicious: slurmctld.service - Slurm controller daemon Loaded: loaded (/lib/systemd/system/slurmctld.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/slurmctld.service.d └─override.conf Active: active (running) since Wed 2024-02-07 15:50:56 UTC; 19min ago Main PID: 51552 (slurmctld) Tasks: 21 (limit: 9363) Memory: 10.4M CPU: 1min 16.088s CGroup: /system.slice/slurmctld.service ├─51552 /usr/sbin/slurmctld --systemd └─51553 "slurmctld: slurmscriptd" "" "" "" "" "" "" Feb 07 15:58:21 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: sched: _slurm_rpc_allocate_resources JobId=3 NodeList=(null) usec=959 Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 WTERMSIG 2 Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 cancelled by interactive user Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 done Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _slurm_rpc_complete_job_allocation: JobId=3 error Job/step already completing or completed Feb 07 15:58:42 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: sched: _slurm_rpc_allocate_resources JobId=4 NodeList=cluster-master-2vt2bqh7ahec04c,cluster-worker-2vt2bqh7ahec04c-2 usec=512 Feb 07 16:06:04 cluster-master-2vt2bqh7ahec04c slurmctld[51553]: slurmctld: error: _run_script: JobId=0 resumeprog exit status 1:0 Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=4 WTERMSIG 2 Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=4 done Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _slurm_rpc_complete_job_allocation: JobId=4 error Job/step already completing or completed I am unsure how to debug this further. It might be coming from a previous problem I tried to fix (basically a few deprecated keys in the configuration). I will try to restart the entire cluster with the added changes to rule out any follow up errors, but maybe it's something obvious a fellow list user can see. Best regards, Xaver

2 2

sinfo gresused shard count wrong/incomplete
by Reed Dier 07 Feb '24

07 Feb '24

I have a bash script that grabs current statistics from sinfo to ship into a time series database to use for Grafana dashboards. We recently began using shards with our gpus, and I’m seeing some unexpected behavior with the data reported from sinfo. > $ sinfo -h -O "NodeHost:5 ,GresUsed:100 ,Gres:100" | grep gpu03 > gpu03 gpu:p40:0(IDX:N/A),gpu:rtx:0(IDX:N/A),shard:p40:1(IDX:1),shard:rtx:0(IDX:N/A) gpu:p40:6(S:0),gpu:rtx:2(S:0),shard:p40:36(S:0),shard:rtx:12(S:0) > > $ scontrol show node gpu03 > NodeName=gpu03 Arch=x86_64 CoresPerSocket=22 > CPUAlloc=52 CPUEfctv=88 CPUTot=88 CPULoad=43.67 > AvailableFeatures=avx,avx512,largeMemory,matlab > ActiveFeatures=avx,avx512,largeMemory,matlab > Gres=gpu:p40:6(S:0),gpu:rtx:2(S:0),shard:p40:36(S:0),shard:rtx:12(S:0) > NodeAddr=gpu03 NodeHostName=gpu03 Version=22.05.8 > OS=Linux 5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023 > RealMemory=768000 AllocMem=425984 FreeMem=116850 Sockets=2 Boards=1 > State=MIXED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A > Partitions=gpu > BootTime=2024-02-02T14:17:07 SlurmdStartTime=2024-02-05T12:33:50 > LastBusyTime=2024-02-07T09:48:50 > CfgTRES=cpu=88,mem=750G,billing=88,gres/gpu=8,gres/gpu:p40=6,gres/gpu:rtx=2,gres/shard=48 > AllocTRES=cpu=52,mem=416G,gres/shard=21 > CapWatts=n/a > CurrentWatts=0 AveWatts=0 > ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s Specifically, we can see there are 48 total shards on this specific node, 36 of type p40, 12 of type rtx. From control, I can see that 21 shards are in use, and from nvidia-smi I can see that the breakdown of shards per gpu index is: [0]p40:4/6 [1]p40:6/6 [2]rtx:4/6 [3]rtx:3/6 [4]p40:4/6 [5]p40:0/6 [6]p40:0/6 [7]p40:0/6 So, I can tell that what “GresUsed” is reporting is actually an entire gpu’s worth of shards used, and the index of that specific entire GPU consumed. However, what it doesn’t show is the amount of specific gres/shard actually used, or more succinctly, I can’t seem to extrapolate that there are 21 shards in use. Right now my script says there is 1 shard out of 48 used, because I’m scraping for the num of shards in use, which is reporting the value of entire GPUs used by shards, and anything less than 100% is rounded to 0.. So, given that I’m on 22.05.8, maybe its better in 23.02.X which I’m hoping to move to within the next month. However I can’t seem to find anything in the release notes for 23.02 or 23.11 that would imply that sinfo reports (my) expected value of the actual count of shards used, Does anyone have any ideas for how I might be able to achieve what I’m looking for using sinfo, or should I instead try to use the sinfo json parser which has a “tres_used” field, that doesn’t appear to be accessible outside of the json output? > { > "architecture": "x86_64", > "burstbuffer_network_address": "", > "boards": 1, > "boot_time": 1706901428, > "comment": "", > "cores": 22, > "cpu_binding": 0, > [SNIP] > "tres": "cpu=88,mem=750G,billing=88,gres/gpu=8,gres/gpu:p40=6,gres/gpu:rtx=2,gres/shard=48", > "slurmd_version": "22.05.8", > "alloc_memory": 284380, > "alloc_cpus": 62, > "idle_cpus": 26, > "tres_used": "cpu=62,mem=284380M,gres/gpu=7,gres/gpu:p40=5,gres/gpu:rtx=2,gres/shard=17", > "tres_weighted": 62 > }, Any ideas appreciated. Thanks, Reed

1 0

Re: Restricting local disk storage of jobs
by Tim Schneider 07 Feb '24

07 Feb '24

Hey Jeffrey, thanks for this suggestion! This is probably the way to go if one can find a way to access GRES in the prolog. I read somewhere that people were calling scontrol to get this information, but this seems a bit unclean. Anyway, if I find some time I will try it out. Best, Tim On 2/6/24 16:30, Jeffrey T Frey wrote: > Most of my ideas have revolved around creating file systems on-the-fly > as part of the job prolog and destroying them in the epilog. The > issue with that mechanism is that formatting a file system (e.g. > mkfs.<type>) can be time-consuming. E.g. formatting your local > scratch SSD as an LVM PV+VG and allocating per-job volumes, you'd > still need to run a e.g. mkfs.xfs and mount the new file system. > > > ZFS file system creation is much quicker (basically combines the LVM + > mkfs steps above) but I don't know of any clusters using ZFS to manage > local file systems on the compute nodes :-) > > > One /could/ leverage XFS project quotas. E.g. for Slurm job 2147483647: > > > *[root@r00n00 /]# mkdir /tmp-alloc/slurm-2147483647* > *[root@r00n00 /]# xfs_quota -x -c 'project -s -p > /tmp-alloc/slurm-2147483647 2147483647' /tmp-alloc* > Setting up project 2147483647 (path /tmp-alloc/slurm-2147483647)... > Processed 1 (/etc/projects and cmdline) paths for project > 2147483647 with recursion depth infinite (-1). > *[root@r00n00 /]# xfs_quota -x -c 'limit -p bhard=1g 2147483647' > /tmp-alloc* > *[root@r00n00 /]# cd /tmp-alloc/slurm-2147483647* > *[root@r00n00 slurm-2147483647]# dd if=/dev/zero of=zeroes bs=5M > count=1000* > dd: error writing ‘zeroes’: No space left on device > 205+0 records in > 204+0 records out > 1073741824 bytes (1.1 GB) copied, 2.92232 s, 367 MB/s > > : > > [root@r00n00 /]# rm -rf /tmp-alloc/*slurm-2147483647* > [root@r00n00 /]# *xfs_quota -x -c 'limit -p bhard=0 2147483647' > /tmp-alloc* > > > Since Slurm jobids max out at 0x03FFFFFF (and 2147483647 = 0x7FFFFFFF) > we have an easy on-demand project id to use on the file system. Slurm > tmpfs plugins have to do a mkdir to create the per-job directory, > adding two xfs_quota commands (which run in more or less O(1) time) > won't extend the prolog by much. Likewise, Slurm tmpfs plugins have to > scrub the directory at job cleanup, so adding another xfs_quota > command will not do much to change their epilog execution times. The > main question is "where does the tmpfs plugin find the quota limit for > the job?" > > > > > >> On Feb 6, 2024, at 08:39, Tim Schneider via slurm-users >> <slurm-users(a)lists.schedmd.com> wrote: >> >> Hi, >> >> In our SLURM cluster, we are using the job_container/tmpfs plugin to >> ensure that each user can use /tmp and it gets cleaned up after them. >> Currently, we are mapping /tmp into the nodes RAM, which means that >> the cgroups make sure that users can only use a certain amount of >> storage inside /tmp. >> >> Now we would like to use of the node's local SSD instead of its RAM >> to hold the files in /tmp. I have seen people define local storage as >> GRES, but I am wondering how to make sure that users do not exceed >> the storage space they requested in a job. Does anyone have an idea >> how to configure local storage as a proper tracked resource? >> >> Thanks a lot in advance! >> >> Best, >> >> Tim >> >> >> -- >> slurm-users mailing list -- slurm-users(a)lists.schedmd.com >> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com >

2 1

scheme for protected GPU jobs from preemption
by Paul Raines 06 Feb '24

06 Feb '24

After using just Fairshare for over a year on our GPU cluster, we have decided it is not working for us for what we really want to achieve among our groups. We have decided to look at preemption. What we want is for users to NOT have a #job/GPU maximum (if they are only person on the cluster they should be able to use it all), but if another user comes to the "full" cluster they should immediately be able to run some jobs. Thus preemption is needed. In our scheme we want * users to have N protected GPU jobs that cannot be preempted where N is the number of GPUs allocated. * N may not be the same for all users. Some priviledged users get more. * jobs pending in the queue will have lower priority dependent on the number of GPUs allocated to running jobs. Maybe doable somehow with PriorityWeightJobSize though not sure how. * Jobs over N are subject to preemption (and requeued if --requeue is given) with shortest running jobs of the user with most unprotected GPUs preempted first. * another complication is we have a variety of different GPUs and users may ask for specific ones which can limit what unprotected GPU jobs are available for preemption My first attempt to do this in SLURM was to just create two partitions, GPU and GPU-req, with different PriorityTier values and the later partition have PreemptMode=REQUEUE. But N would be set by a MaxTRES on the first partition and be the same for everyone and we need it to be INDEPENDENT for each user. Also users would have to "think" about which partition to submit jobs to. And users want their longest running "unprotected" job to be able to be PROMOTED to a "protected" jobs automatically when a "protected" job finishes. However slurm does not allow running jobs to move between partitions. I am trying to figure out QOS pre-emption which might solve the independent N per user issue but I don't think it will solve the promotion issue. Any ideas how this scheme might be possible in SLURM? Otherwise I might have to write a complicated cron job that tries to do it all "outside" of SLURM issuing scontrol commands. --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

Restricting local disk storage of jobs
by Tim Schneider 06 Feb '24

06 Feb '24

Hi, In our SLURM cluster, we are using the job_container/tmpfs plugin to ensure that each user can use /tmp and it gets cleaned up after them. Currently, we are mapping /tmp into the nodes RAM, which means that the cgroups make sure that users can only use a certain amount of storage inside /tmp. Now we would like to use of the node's local SSD instead of its RAM to hold the files in /tmp. I have seen people define local storage as GRES, but I am wondering how to make sure that users do not exceed the storage space they requested in a job. Does anyone have an idea how to configure local storage as a proper tracked resource? Thanks a lot in advance! Best, Tim

3 2

Re: [ext] Restricting local disk storage of jobs
by Tim Schneider 06 Feb '24

06 Feb '24

Hi Magnus, I understand. Thanks a lot for your suggestion. Best, Tim On 06.02.24 15:34, Hagdorn, Magnus Karl Moritz wrote: > Hi Tim, > in the end the InitScript didn't contain anything useful because > > slurmd: error: _parse_next_key: Parsing error at unrecognized key: > InitScript > > At this stage I gave up. This was with SLURM 23.02. My plan was to > setup the local scratch directory with XFS and then get the script to > apply a project quota, ie quota attached to the directory. > > I would start by checking if slurm recognises the InitScript option. > > Regards > magnus > > On Tue, 2024-02-06 at 15:24 +0100, Tim Schneider wrote: >> Hi Magnus, >> >> thanks for your reply! If you can, would you mind sharing the >> InitScript >> of your attempt at getting it to work? >> >> Best, >> >> Tim >> >> On 06.02.24 15:19, Hagdorn, Magnus Karl Moritz wrote: >>> Hi Tim, >>> we are using the container/tmpfs plugin to map /tmp to a local NVMe >>> drive which works great. I did consider setting up directory >>> quotas. I >>> thought the InitScript [1] option should do the trick. Alas, I >>> didn't >>> get it to work. If I remember correctly, slurm complained about the >>> option being present. In the end we recommend our users to make >>> exclusive use a node if they are going to use a lot of local >>> scratch >>> space. I don't think this happens very often if at all. >>> Regards >>> magnus >>> >>> [1] >>> https://slurm.schedmd.com/job_container.conf.html#OPT_InitScript >>> >>> >>> On Tue, 2024-02-06 at 14:39 +0100, Tim Schneider via slurm-users >>> wrote: >>>> Hi, >>>> >>>> In our SLURM cluster, we are using the job_container/tmpfs plugin >>>> to >>>> ensure that each user can use /tmp and it gets cleaned up after >>>> them. >>>> Currently, we are mapping /tmp into the nodes RAM, which means >>>> that >>>> the >>>> cgroups make sure that users can only use a certain amount of >>>> storage >>>> inside /tmp. >>>> >>>> Now we would like to use of the node's local SSD instead of its >>>> RAM >>>> to >>>> hold the files in /tmp. I have seen people define local storage >>>> as >>>> GRES, >>>> but I am wondering how to make sure that users do not exceed the >>>> storage >>>> space they requested in a job. Does anyone have an idea how to >>>> configure >>>> local storage as a proper tracked resource? >>>> >>>> Thanks a lot in advance! >>>> >>>> Best, >>>> >>>> Tim >>>> >>>>

1 0

Re: [ext] Restricting local disk storage of jobs
by Tim Schneider 06 Feb '24

06 Feb '24

Hi Magnus, thanks for your reply! If you can, would you mind sharing the InitScript of your attempt at getting it to work? Best, Tim On 06.02.24 15:19, Hagdorn, Magnus Karl Moritz wrote: > Hi Tim, > we are using the container/tmpfs plugin to map /tmp to a local NVMe > drive which works great. I did consider setting up directory quotas. I > thought the InitScript [1] option should do the trick. Alas, I didn't > get it to work. If I remember correctly, slurm complained about the > option being present. In the end we recommend our users to make > exclusive use a node if they are going to use a lot of local scratch > space. I don't think this happens very often if at all. > Regards > magnus > > [1] > https://slurm.schedmd.com/job_container.conf.html#OPT_InitScript > > > On Tue, 2024-02-06 at 14:39 +0100, Tim Schneider via slurm-users wrote: >> Hi, >> >> In our SLURM cluster, we are using the job_container/tmpfs plugin to >> ensure that each user can use /tmp and it gets cleaned up after them. >> Currently, we are mapping /tmp into the nodes RAM, which means that >> the >> cgroups make sure that users can only use a certain amount of storage >> inside /tmp. >> >> Now we would like to use of the node's local SSD instead of its RAM >> to >> hold the files in /tmp. I have seen people define local storage as >> GRES, >> but I am wondering how to make sure that users do not exceed the >> storage >> space they requested in a job. Does anyone have an idea how to >> configure >> local storage as a proper tracked resource? >> >> Thanks a lot in advance! >> >> Best, >> >> Tim >> >>

2 1

Starting a job after a file is created in previous job (dependency looking for soluton)
by Amjad Syed 06 Feb '24

06 Feb '24

Hello I have the following scenario: I need to submit a sequence of up to 400 jobs where the even jobs depend on the preceeding odd job to finish and every odd job depends on the presence of a file generated by the preceding even job (availability of the file for the first of those 400 jobs is guaranteed). If I just submit all those jobs via a loop using dependencies, then I end up with a lot of pending jobs who might later not even run because no output file has been produced by the preceding jobs. Is there a way to pause the submission loop until the required file has been generated so that at most two jobs are submitted at the same time? Here is a sample submission script showing what I want to achieve. for i in {1..200}; do FILE=GHM_paramset_${i}.dat # How can I pause the submission loop until the FILE has been created???? #if test -f "$FILE"; then jobid4=$(sbatch --parsable --dependency=afterok:$jobid3 job4_sub $i) jobid3=$(sbatch --parsable --dependency=afterok:$jobid4 job3_sub $i) #fi done Any help will be appreciated Amjad

3 2

SLURM configuration for LDAP users
by Richard Chang 05 Feb '24

05 Feb '24

Hi, I am a little new to this, so please pardon my ignorance. I have configured slurm in my cluster and it works fine with local users. But I am not able to get it working with LDAP/SSSD authentication. User logins using ssh are working fine. An LDAP user can login to the login, slurmctld and compute nodes, but when they try to submit jobs, slurmctld logs an error about invalid account or partition for user. Someone said we need to add the user manually into the database using the sacctmgr command. But I am not sure we need to do this for each and every LDAP user. Yes, it does work if we add the LDAP user manually using sacctmgr. But I am not convinced this manual way is the way to do. The documentation is not very clear about using LDAP accounts. Saw somewhere in the list about using UsePAM=1 and copying or creating a softlink for slurm PAM module under /etc/pam.d . But it didn't work for me. Saw somewhere else that we need to specifying LaunchParameters=enable_nss_slurm in the slurm.conf file and put slurm keyword in passwd/group entry in the /etc/nsswitch.conf file. Did these, but didn't help either. I am bereft of ideas at present. If anyone has real world experience and can advise, I will be grateful. Thank you, Richard

4 4