February 2024 - slurm-users - lists.schedmd.com

Increasing SlurmdTimeout beyond 300 Seconds
by Andrew Baughan (ITCS - Staff) 12 Feb '24

12 Feb '24

Hi, We've been experiencing issues with network saturation on our older nodes caused by storage (GPFS) backups. This causes slurmctld to loose contact with slurmd on some compute nodes and for user jobs to be killed. While the longer term solution is to replace these and upgrade the network, I'm wondering if there are any ramifications, beyond nodes with genuine issues taking longer to get marked down, by increasing SlurmdTimeout. We've already applied a modest increase which has helped but not resolved the issue and wondering if we should push it further in the interim. Kind Regards Andy Baughan HPC Systems Developer

4 3

simple question, I guess… from a newbie sysadmin
by Richard Randriatoamanana 12 Feb '24

12 Feb '24

Hi, I am trying to help a sysadmin colleague (and to understand for myself) trying to configure a new slurm server and he struggles to understand if there is an alternative way to config slurm managing job policy submission per user without necessarily installing an accounting mariadb service. I said no but maybe I misunderstand slurm admin configuration. Thanks in advance for feedback. Bests, Richard — Sent from my rotary and brandy phone ! Apologies for typos and any spelling errors

2 1

Question on using a suffix with "squeue --Format=FORMAT"
by urbanjost 10 Feb '24

10 Feb '24

I do not see how to use the suffix for the --Format option of the squeue(1) command. The description does not indicate how to tell the end of the size and period components and the beginning of the suffix, but assuming the suffix must being with something other than a period or numeric character I found all suffix values produce a null separator. A wanted to use --Format somewhat like --format=%all, producing all fields. As the --Format did not have an equivalent descriptor, (ie. something like --Format=All ) it looked like it would be required to list all the allowed keywords with a suffix of "|", which from the description I assumed would be as simple as a longer version of something like thi: squeue --Format='STDOUT:|,STDERR:|,STDIN:|,WorkDir:|' Given the description: -O, --Format=<output_format> Specify the information to be displayed. Also see the -o, --format=<output_format> option described above (which supports greater flexibility in formatting, but does not support access to all fields because we ran out of letters). Requests a comma separated list of job information to be displayed. The format of each field is "type[:[.][size][suffix]]" size Minimum field size. If no size is specified, 20 char‐ acters will be allocated to print the information. . Indicates the output should be right justified and size must be specified. By default output is left justified. suffix Arbitrary string to append to the end of the field. Note that many of these type specifications are valid only for jobs while others are valid only for job steps. Valid type specifications include: Even something like this produces no error but the suffix is always a null string: squeue --Format='STDOUT:SUFFIX,STDERR:SUFFIXSTDIN:SUFFIXWorkDir:SUFFIX So if I want to generate output with squeue --Format=FORMAT with a vertical line character between the output fields what is the proper syntax? A test bash(1) script where I could not get the SUFFIX variable to work as expected ... #!/bin/bash #@(#) Build value for squeue --Format='FORMAT' that includes all available fields with a vertical bar separating each field GOODNAMES='' BADNAMES='' SUFFIX='SUFFIX' GOODNAMES='' BADNAMES='' SUFFIX='1000:|' GOODNAMES='' BADNAMES='' SUFFIX='|' for NAME in STDOUT Account AccrueTime admin_comment AllocNodes AllocSID ArrayJobID ArrayTaskID AssocID BatchFlag BatchHost BoardsPerNode BurstBuffer BurstBufferState Cluster ClusterFeature Command Comment Container ContainerID Contiguous Cores CoreSpec CPUFreq cpus-per-task cpus-per-tres Deadline DelayBal Licenses MaxCPUs MaxNodes MCSLabel mem-per-tres MinCpus MinMemory MinTime Mintition PendingTime PreemptTime Prefer Priority PriorityLong Profile QOS Reason Riable SiblingsViableRaw Sockets SPerBoard StartTime State StateCompact STDERR STres-per-node tres-per-socket tres-per-step tres-per-task UserID UserName Wait4Sw do if squeue --Format="$NAME" >/dev/null then GOODNAMES="$GOODNAMES${NAME}:${SUFFIX}," else BADNAMES="$BADNAMES${NAME}," fi done 2>&1 #FORMAT=$(tr --squeeze-repeats ' ' ' '<<<$GOODNAMES) FORMAT=${GOODNAMES%%,} # trim off trailing comma read r c < <(stty size) # get screen width for use with cut(1) # try it squeue --Format="$FORMAT"|column -t|cut -c 1-$c # echo it echo "$FORMAT" exit Sent with [Proton Mail](https://proton.me/) secure email.

1 0

question on sbatch --prefer
by Alan Stange 10 Feb '24

10 Feb '24

Hello all, I'm somewhat new to Slurm, but long time user of other batch systems. Assume we have a simple cluster of uniform racks of systems with no special resources, and our jobs are all single cpu tasks. Lets say I have a long running job in the cluster, which needs to spawn a helper process into the cluster. We have a strong preference for this helper to run on the same cluster node as the original job, but if that node is already scheduled full, then we want this new task to be scheduled on another systems without any delay. The problem I have is that the --nodelist doesn't solve this, and, as far as I can tell, there's no option with --prefer to specify a node name as a resource, without creating a gres for every hostname in the cluster. It seems like what I'm trying to do should be achievable, but having read though the documentation and searched the archives of this list, I'm not seeing a solution. I'm hoping someone here has some experience with this and can point me in the right direction. Sincerely, Alan

3 3

Compilation question
by Sylvain MARET 10 Feb '24

10 Feb '24

Hello everyone ! I'm trying to compile slurm 22.05.11 on Rocky linux 8.7 with freeipmi support I've seen the documentation so I've done the configure step : ./configure --with-pmix=$PMIXHOME --with-ucx=$UCXHOME --with-nvml=$NVMLHOME --prefix=$SLURMHOME --with-freeipmi=/usr but when I run make I end up with the following error : /bin/sh ../../../../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -L/usr/lib64 -lhdf5_hl -lhdf5 -lsz -lz -ldl -lm -o sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -L../../../../../src/api/.libs -lslurmfull -ldl ../libhdf5_api.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o .libs/sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -Wl,--export-dynamic -L/usr/lib64 -L../../../../../src/api/.libs /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so ../.libs/libhdf5_api.a -lhdf5_hl -lhdf5 -lsz -lz -ldl -lpthread -lm -lresolv -pthread -Wl,-rpath -Wl,/softs/batch/slurm/22.05.11/lib/slurm sh5util.o:(.init_array+0x0): undefined reference to `__nv_init_env' sh5util.o:(.init_array+0x8): undefined reference to `__flushz' sh5util.o:(.init_array+0x10): undefined reference to `__daz' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_transfer_unique' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_sort_key_pairs' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrchr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_unsetenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_sort' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_for_each' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin__pgi_isnanld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_extra_conf_path' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_ctzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_running_in_slurmctld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mcopy1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_clzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_count' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin_va_gparg1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_destroy_config_key_pair' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xfree_ptr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_getenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_free_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_log_level' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mset8' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup_printf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_delete_first' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_append' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_error' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_init_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_log_var' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_parse_file' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_fatal' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_popcountll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrcmp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_destroy' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_setenvpf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_destroy' collect2: error: ld returned 1 exit status What can I do to resolve these undefined reference error ? Regards, Sylvain Maret

3 2

Re: Selecting only a subset of GPU's from all available GPU's
by Minulakshmi S 10 Feb '24

10 Feb '24

Hi Loris, I have different kind of GPU's in same node , and I beleive feature applies to particular node , and cant be applied to only a few GPU's connected to a single node. On Mon, 18 Dec 2023, 12:14 Loris Bennett, <loris.bennett(a)fu-berlin.de> wrote: > Hi Minu, > > Minulakshmi S <minulakshmi.s(a)gmail.com> writes: > > > I'm submitting jobs to a cluster via SLURM scheduler, and let's say I > have access to 8 GPUs in my cluster in same node. They are GPUs of type > A,B,C,D,E,F,G,H. I > > would like to submit a job that requests the use of GPUs of type A or B > or C but NOT of type D/E/F/G/H. So I need some type of OR logic with the > --gres flag. > > > > Eg : When I request GPU of type A , I can do sbatch –gres=gpu:TypeA:1, > I need to input a subset of GPU’s and let slurm schedule job utilizing one > of the GPU > > from this allowed list. > > > > Regards > > Minu > > Assuming the GPUs within a node are all of the same type, could you define > a feature for each GPU type, assign the features to the appropriate > nodes, and then run the job with > > --contraint=gpu_TypeA --gres=gpu:1 > > ? > > This is obviously rather clunky and it would be much nicer if > multiple GPUs types passed to '--gres' were ORed. > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin >

1 0

Memory used per node
by Gerhard Strangar 09 Feb '24

09 Feb '24

Hello, I'm wondering if there's a way to tell how much memory my job is using per node. I'm doing #SBATCH -n 256 srun solver inputfile When I run sacct -o maxvmsize, the result apparently is the maxmimum VSZ of the largest solver process, not the maximum of the sum of them all (unlike when calling mpirun instead). When I sstat -o TresUsageInMax, I get the memory summed up over all nodes being used. Can I get the maximum VSZ per node? Gerhard

2 1

Could not find group with gid even when they exist
by Nic Lewis 09 Feb '24

09 Feb '24

After upgrading to version 23.11.3 we started to get slammed with the following log messages from slurmctld "error: validate_group: Could not find group with gid <id>" This spans a handful of groups and repeats constantly, drowning out just about everything else. Attempting to do a lookup on the group shows that they exist on the scheduler node, same for all the submission and compute nodes. As far as I can tell, slurm should be able to locate the group in question. Jobs submitted from users within those groups go through just fine. They get scheduled, run, and clean up no problem. I'm at a loss on where to look next.

1 1

Relative QOS limits in sacctmgr?
by Chip Seraphine 09 Feb '24

09 Feb '24

Hello, TL,DR: How does the relative QOS flag work? I have a QOS and I want it to be collectively restricted to 50% of the reachable cores in the cluster. I’ve been managing this by dividing my core count to 2 to get N, and doing ‘sacctmgr update qos foobar set MaxTRES=cpu=N’. That’s fine, except that N changes frequently as worker nodes come and go, so I have an hourly cronjob that enforces this. The problem with *that* is the values you back out with ‘sacctmgr show qos’ are often in different units than you input (e.g. gigs instead of megs of ram), which means I need to do some annoying parsing, will miss minor rounding errors, etc. (Using JSON solves that problem but introduces others that I do not want to deal with.) So while looking for a better way I noticed that QOSs support a flag called “relative”, which says “the QOS limits will be treated as percentages of a cluster/partition instead of absolutes”. Perfect! However, it’s not very clear to me what that means in practice. For example, with the “relative” flag set, the MaxTRES units still seem to work the same, expect metric suffices (“G”, ”M”, etc) and won’t accept periods. Is anyone actually using this flag, and if so can you tell me what attributes it affects, and in what way? -- Chip Seraphine Grid Operations For support please use help-grid in email or slack. This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

1 0

Is there a way to list allocated/unallocated resources defined in a QoS?
by Alastair Neil 09 Feb '24

09 Feb '24

Slurm version 23.02.07 If I have a QoS defined that has a set number of say GPU devices set in the GrpTRES. Is there an easy way to generate a list of how much of the defined quota is allocated or conversely un-allocated? e.g.: Name|Priority|GraceTime|Preempt|PreemptExemptTime|PreemptMode|Flags|UsageThres|UsageFactor|GrpTRES|GrpTRESMins|GrpTRESRunMins|GrpJobs|GrpSubmit|GrpWall|MaxTRES|MaxTRESPerNode|MaxTRESMins|MaxWall|MaxTRESPU|MaxJobsPU|MaxSubmitPU|MaxTRESPA|MaxJobsPA|MaxSubmitPA|MinTRES| normal|0|00:00:00|||cluster|||1.000000|||||||||||cpu=3000,gres/gpu=20||||||| dept1|1|00:00:00|||cluster|||1.000000|cpu=256,gres/gpu:1g.10gb=16,gres/gpu:2g.20gb=8,gres/gpu:3g.40gb=8,gres/gpu:a100.80gb=8||||||||||||||||| dept2|1|00:00:00|||cluster|||1.000000|cpu=256,gres/gpu:1g.10gb=0,gres/gpu:2g.20gb=0,gres/gpu:3g.40gb=0,gres/gpu:a100.80gb=16||||||||||||||||| So dept1 and dept2 qos are set on the same partition. How can a user with access to one or other see if there are available resources in the partition?

2 1

2024

slurm-users February 2024 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users February 2024