- slurm-users - lists.schedmd.com

Question on using a suffix with "squeue --Format=FORMAT"
by urbanjost 11 Feb '24

11 Feb '24

I do not see how to use the suffix for the --Format option of the squeue(1) command. The description does not indicate how to tell the end of the size and period components and the beginning of the suffix, but assuming the suffix must being with something other than a period or numeric character I found all suffix values produce a null separator. A wanted to use --Format somewhat like --format=%all, producing all fields. As the --Format did not have an equivalent descriptor, (ie. something like --Format=All ) it looked like it would be required to list all the allowed keywords with a suffix of "|", which from the description I assumed would be as simple as a longer version of something like thi: squeue --Format='STDOUT:|,STDERR:|,STDIN:|,WorkDir:|' Given the description: -O, --Format=<output_format> Specify the information to be displayed. Also see the -o, --format=<output_format> option described above (which supports greater flexibility in formatting, but does not support access to all fields because we ran out of letters). Requests a comma separated list of job information to be displayed. The format of each field is "type[:[.][size][suffix]]" size Minimum field size. If no size is specified, 20 char‐ acters will be allocated to print the information. . Indicates the output should be right justified and size must be specified. By default output is left justified. suffix Arbitrary string to append to the end of the field. Note that many of these type specifications are valid only for jobs while others are valid only for job steps. Valid type specifications include: Even something like this produces no error but the suffix is always a null string: squeue --Format='STDOUT:SUFFIX,STDERR:SUFFIXSTDIN:SUFFIXWorkDir:SUFFIX So if I want to generate output with squeue --Format=FORMAT with a vertical line character between the output fields what is the proper syntax? A test bash(1) script where I could not get the SUFFIX variable to work as expected ... #!/bin/bash #@(#) Build value for squeue --Format='FORMAT' that includes all available fields with a vertical bar separating each field GOODNAMES='' BADNAMES='' SUFFIX='SUFFIX' GOODNAMES='' BADNAMES='' SUFFIX='1000:|' GOODNAMES='' BADNAMES='' SUFFIX='|' for NAME in STDOUT Account AccrueTime admin_comment AllocNodes AllocSID ArrayJobID ArrayTaskID AssocID BatchFlag BatchHost BoardsPerNode BurstBuffer BurstBufferState Cluster ClusterFeature Command Comment Container ContainerID Contiguous Cores CoreSpec CPUFreq cpus-per-task cpus-per-tres Deadline DelayBal Licenses MaxCPUs MaxNodes MCSLabel mem-per-tres MinCpus MinMemory MinTime Mintition PendingTime PreemptTime Prefer Priority PriorityLong Profile QOS Reason Riable SiblingsViableRaw Sockets SPerBoard StartTime State StateCompact STDERR STres-per-node tres-per-socket tres-per-step tres-per-task UserID UserName Wait4Sw do if squeue --Format="$NAME" >/dev/null then GOODNAMES="$GOODNAMES${NAME}:${SUFFIX}," else BADNAMES="$BADNAMES${NAME}," fi done 2>&1 #FORMAT=$(tr --squeeze-repeats ' ' ' '<<<$GOODNAMES) FORMAT=${GOODNAMES%%,} # trim off trailing comma read r c < <(stty size) # get screen width for use with cut(1) # try it squeue --Format="$FORMAT"|column -t|cut -c 1-$c # echo it echo "$FORMAT" exit Sent with [Proton Mail](https://proton.me/) secure email.

1 0

question on sbatch --prefer
by Alan Stange 10 Feb '24

10 Feb '24

Hello all, I'm somewhat new to Slurm, but long time user of other batch systems. Assume we have a simple cluster of uniform racks of systems with no special resources, and our jobs are all single cpu tasks. Lets say I have a long running job in the cluster, which needs to spawn a helper process into the cluster. We have a strong preference for this helper to run on the same cluster node as the original job, but if that node is already scheduled full, then we want this new task to be scheduled on another systems without any delay. The problem I have is that the --nodelist doesn't solve this, and, as far as I can tell, there's no option with --prefer to specify a node name as a resource, without creating a gres for every hostname in the cluster. It seems like what I'm trying to do should be achievable, but having read though the documentation and searched the archives of this list, I'm not seeing a solution. I'm hoping someone here has some experience with this and can point me in the right direction. Sincerely, Alan

3 3

Compilation question
by Sylvain MARET 10 Feb '24

10 Feb '24

Hello everyone ! I'm trying to compile slurm 22.05.11 on Rocky linux 8.7 with freeipmi support I've seen the documentation so I've done the configure step : ./configure --with-pmix=$PMIXHOME --with-ucx=$UCXHOME --with-nvml=$NVMLHOME --prefix=$SLURMHOME --with-freeipmi=/usr but when I run make I end up with the following error : /bin/sh ../../../../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -L/usr/lib64 -lhdf5_hl -lhdf5 -lsz -lz -ldl -lm -o sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -L../../../../../src/api/.libs -lslurmfull -ldl ../libhdf5_api.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o .libs/sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -Wl,--export-dynamic -L/usr/lib64 -L../../../../../src/api/.libs /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so ../.libs/libhdf5_api.a -lhdf5_hl -lhdf5 -lsz -lz -ldl -lpthread -lm -lresolv -pthread -Wl,-rpath -Wl,/softs/batch/slurm/22.05.11/lib/slurm sh5util.o:(.init_array+0x0): undefined reference to `__nv_init_env' sh5util.o:(.init_array+0x8): undefined reference to `__flushz' sh5util.o:(.init_array+0x10): undefined reference to `__daz' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_transfer_unique' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_sort_key_pairs' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrchr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_unsetenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_sort' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_for_each' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin__pgi_isnanld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_extra_conf_path' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_ctzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_running_in_slurmctld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mcopy1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_clzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_count' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin_va_gparg1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_destroy_config_key_pair' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xfree_ptr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_getenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_free_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_log_level' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mset8' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup_printf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_delete_first' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_append' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_error' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_init_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_log_var' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_parse_file' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_fatal' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_popcountll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrcmp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_destroy' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_setenvpf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_destroy' collect2: error: ld returned 1 exit status What can I do to resolve these undefined reference error ? Regards, Sylvain Maret

3 2

Re: Selecting only a subset of GPU's from all available GPU's
by Minulakshmi S 10 Feb '24

10 Feb '24

Hi Loris, I have different kind of GPU's in same node , and I beleive feature applies to particular node , and cant be applied to only a few GPU's connected to a single node. On Mon, 18 Dec 2023, 12:14 Loris Bennett, <loris.bennett(a)fu-berlin.de> wrote: > Hi Minu, > > Minulakshmi S <minulakshmi.s(a)gmail.com> writes: > > > I'm submitting jobs to a cluster via SLURM scheduler, and let's say I > have access to 8 GPUs in my cluster in same node. They are GPUs of type > A,B,C,D,E,F,G,H. I > > would like to submit a job that requests the use of GPUs of type A or B > or C but NOT of type D/E/F/G/H. So I need some type of OR logic with the > --gres flag. > > > > Eg : When I request GPU of type A , I can do sbatch –gres=gpu:TypeA:1, > I need to input a subset of GPU’s and let slurm schedule job utilizing one > of the GPU > > from this allowed list. > > > > Regards > > Minu > > Assuming the GPUs within a node are all of the same type, could you define > a feature for each GPU type, assign the features to the appropriate > nodes, and then run the job with > > --contraint=gpu_TypeA --gres=gpu:1 > > ? > > This is obviously rather clunky and it would be much nicer if > multiple GPUs types passed to '--gres' were ORed. > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin >

1 0

Memory used per node
by Gerhard Strangar 09 Feb '24

09 Feb '24

Hello, I'm wondering if there's a way to tell how much memory my job is using per node. I'm doing #SBATCH -n 256 srun solver inputfile When I run sacct -o maxvmsize, the result apparently is the maxmimum VSZ of the largest solver process, not the maximum of the sum of them all (unlike when calling mpirun instead). When I sstat -o TresUsageInMax, I get the memory summed up over all nodes being used. Can I get the maximum VSZ per node? Gerhard

2 1

Could not find group with gid even when they exist
by Nic Lewis 09 Feb '24

09 Feb '24

After upgrading to version 23.11.3 we started to get slammed with the following log messages from slurmctld "error: validate_group: Could not find group with gid <id>" This spans a handful of groups and repeats constantly, drowning out just about everything else. Attempting to do a lookup on the group shows that they exist on the scheduler node, same for all the submission and compute nodes. As far as I can tell, slurm should be able to locate the group in question. Jobs submitted from users within those groups go through just fine. They get scheduled, run, and clean up no problem. I'm at a loss on where to look next.

1 1

Relative QOS limits in sacctmgr?
by Chip Seraphine 09 Feb '24

09 Feb '24

Hello, TL,DR: How does the relative QOS flag work? I have a QOS and I want it to be collectively restricted to 50% of the reachable cores in the cluster. I’ve been managing this by dividing my core count to 2 to get N, and doing ‘sacctmgr update qos foobar set MaxTRES=cpu=N’. That’s fine, except that N changes frequently as worker nodes come and go, so I have an hourly cronjob that enforces this. The problem with *that* is the values you back out with ‘sacctmgr show qos’ are often in different units than you input (e.g. gigs instead of megs of ram), which means I need to do some annoying parsing, will miss minor rounding errors, etc. (Using JSON solves that problem but introduces others that I do not want to deal with.) So while looking for a better way I noticed that QOSs support a flag called “relative”, which says “the QOS limits will be treated as percentages of a cluster/partition instead of absolutes”. Perfect! However, it’s not very clear to me what that means in practice. For example, with the “relative” flag set, the MaxTRES units still seem to work the same, expect metric suffices (“G”, ”M”, etc) and won’t accept periods. Is anyone actually using this flag, and if so can you tell me what attributes it affects, and in what way? -- Chip Seraphine Grid Operations For support please use help-grid in email or slack. This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

1 0

Is there a way to list allocated/unallocated resources defined in a QoS?
by Alastair Neil 09 Feb '24

09 Feb '24

Slurm version 23.02.07 If I have a QoS defined that has a set number of say GPU devices set in the GrpTRES. Is there an easy way to generate a list of how much of the defined quota is allocated or conversely un-allocated? e.g.: Name|Priority|GraceTime|Preempt|PreemptExemptTime|PreemptMode|Flags|UsageThres|UsageFactor|GrpTRES|GrpTRESMins|GrpTRESRunMins|GrpJobs|GrpSubmit|GrpWall|MaxTRES|MaxTRESPerNode|MaxTRESMins|MaxWall|MaxTRESPU|MaxJobsPU|MaxSubmitPU|MaxTRESPA|MaxJobsPA|MaxSubmitPA|MinTRES| normal|0|00:00:00|||cluster|||1.000000|||||||||||cpu=3000,gres/gpu=20||||||| dept1|1|00:00:00|||cluster|||1.000000|cpu=256,gres/gpu:1g.10gb=16,gres/gpu:2g.20gb=8,gres/gpu:3g.40gb=8,gres/gpu:a100.80gb=8||||||||||||||||| dept2|1|00:00:00|||cluster|||1.000000|cpu=256,gres/gpu:1g.10gb=0,gres/gpu:2g.20gb=0,gres/gpu:3g.40gb=0,gres/gpu:a100.80gb=16||||||||||||||||| So dept1 and dept2 qos are set on the same partition. How can a user with access to one or other see if there are available resources in the partition?

2 1

Errors upgrading to 23.11.0
by Xaver Stiensmeier 08 Feb '24

08 Feb '24

Dear slurm-user list, I got this error: Unable to start service slurmctld: Job for slurmctld.service failed because the control process exited with error code.\nSee \"systemctl status slurmctld.service\" and \"journalctl -xeu slurmctld.service\" for details. but in slurmctld.service I see nothing suspicious: slurmctld.service - Slurm controller daemon Loaded: loaded (/lib/systemd/system/slurmctld.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/slurmctld.service.d └─override.conf Active: active (running) since Wed 2024-02-07 15:50:56 UTC; 19min ago Main PID: 51552 (slurmctld) Tasks: 21 (limit: 9363) Memory: 10.4M CPU: 1min 16.088s CGroup: /system.slice/slurmctld.service ├─51552 /usr/sbin/slurmctld --systemd └─51553 "slurmctld: slurmscriptd" "" "" "" "" "" "" Feb 07 15:58:21 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: sched: _slurm_rpc_allocate_resources JobId=3 NodeList=(null) usec=959 Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 WTERMSIG 2 Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 cancelled by interactive user Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 done Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _slurm_rpc_complete_job_allocation: JobId=3 error Job/step already completing or completed Feb 07 15:58:42 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: sched: _slurm_rpc_allocate_resources JobId=4 NodeList=cluster-master-2vt2bqh7ahec04c,cluster-worker-2vt2bqh7ahec04c-2 usec=512 Feb 07 16:06:04 cluster-master-2vt2bqh7ahec04c slurmctld[51553]: slurmctld: error: _run_script: JobId=0 resumeprog exit status 1:0 Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=4 WTERMSIG 2 Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=4 done Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _slurm_rpc_complete_job_allocation: JobId=4 error Job/step already completing or completed I am unsure how to debug this further. It might be coming from a previous problem I tried to fix (basically a few deprecated keys in the configuration). I will try to restart the entire cluster with the added changes to rule out any follow up errors, but maybe it's something obvious a fellow list user can see. Best regards, Xaver

2 2

sinfo gresused shard count wrong/incomplete
by Reed Dier 07 Feb '24

07 Feb '24

I have a bash script that grabs current statistics from sinfo to ship into a time series database to use for Grafana dashboards. We recently began using shards with our gpus, and I’m seeing some unexpected behavior with the data reported from sinfo. > $ sinfo -h -O "NodeHost:5 ,GresUsed:100 ,Gres:100" | grep gpu03 > gpu03 gpu:p40:0(IDX:N/A),gpu:rtx:0(IDX:N/A),shard:p40:1(IDX:1),shard:rtx:0(IDX:N/A) gpu:p40:6(S:0),gpu:rtx:2(S:0),shard:p40:36(S:0),shard:rtx:12(S:0) > > $ scontrol show node gpu03 > NodeName=gpu03 Arch=x86_64 CoresPerSocket=22 > CPUAlloc=52 CPUEfctv=88 CPUTot=88 CPULoad=43.67 > AvailableFeatures=avx,avx512,largeMemory,matlab > ActiveFeatures=avx,avx512,largeMemory,matlab > Gres=gpu:p40:6(S:0),gpu:rtx:2(S:0),shard:p40:36(S:0),shard:rtx:12(S:0) > NodeAddr=gpu03 NodeHostName=gpu03 Version=22.05.8 > OS=Linux 5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023 > RealMemory=768000 AllocMem=425984 FreeMem=116850 Sockets=2 Boards=1 > State=MIXED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A > Partitions=gpu > BootTime=2024-02-02T14:17:07 SlurmdStartTime=2024-02-05T12:33:50 > LastBusyTime=2024-02-07T09:48:50 > CfgTRES=cpu=88,mem=750G,billing=88,gres/gpu=8,gres/gpu:p40=6,gres/gpu:rtx=2,gres/shard=48 > AllocTRES=cpu=52,mem=416G,gres/shard=21 > CapWatts=n/a > CurrentWatts=0 AveWatts=0 > ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s Specifically, we can see there are 48 total shards on this specific node, 36 of type p40, 12 of type rtx. From control, I can see that 21 shards are in use, and from nvidia-smi I can see that the breakdown of shards per gpu index is: [0]p40:4/6 [1]p40:6/6 [2]rtx:4/6 [3]rtx:3/6 [4]p40:4/6 [5]p40:0/6 [6]p40:0/6 [7]p40:0/6 So, I can tell that what “GresUsed” is reporting is actually an entire gpu’s worth of shards used, and the index of that specific entire GPU consumed. However, what it doesn’t show is the amount of specific gres/shard actually used, or more succinctly, I can’t seem to extrapolate that there are 21 shards in use. Right now my script says there is 1 shard out of 48 used, because I’m scraping for the num of shards in use, which is reporting the value of entire GPUs used by shards, and anything less than 100% is rounded to 0.. So, given that I’m on 22.05.8, maybe its better in 23.02.X which I’m hoping to move to within the next month. However I can’t seem to find anything in the release notes for 23.02 or 23.11 that would imply that sinfo reports (my) expected value of the actual count of shards used, Does anyone have any ideas for how I might be able to achieve what I’m looking for using sinfo, or should I instead try to use the sinfo json parser which has a “tres_used” field, that doesn’t appear to be accessible outside of the json output? > { > "architecture": "x86_64", > "burstbuffer_network_address": "", > "boards": 1, > "boot_time": 1706901428, > "comment": "", > "cores": 22, > "cpu_binding": 0, > [SNIP] > "tres": "cpu=88,mem=750G,billing=88,gres/gpu=8,gres/gpu:p40=6,gres/gpu:rtx=2,gres/shard=48", > "slurmd_version": "22.05.8", > "alloc_memory": 284380, > "alloc_cpus": 62, > "idle_cpus": 26, > "tres_used": "cpu=62,mem=284380M,gres/gpu=7,gres/gpu:p40=5,gres/gpu:rtx=2,gres/shard=17", > "tres_weighted": 62 > }, Any ideas appreciated. Thanks, Reed

1 0

Re: Restricting local disk storage of jobs
by Tim Schneider 07 Feb '24

07 Feb '24

Hey Jeffrey, thanks for this suggestion! This is probably the way to go if one can find a way to access GRES in the prolog. I read somewhere that people were calling scontrol to get this information, but this seems a bit unclean. Anyway, if I find some time I will try it out. Best, Tim On 2/6/24 16:30, Jeffrey T Frey wrote: > Most of my ideas have revolved around creating file systems on-the-fly > as part of the job prolog and destroying them in the epilog. The > issue with that mechanism is that formatting a file system (e.g. > mkfs.<type>) can be time-consuming. E.g. formatting your local > scratch SSD as an LVM PV+VG and allocating per-job volumes, you'd > still need to run a e.g. mkfs.xfs and mount the new file system. > > > ZFS file system creation is much quicker (basically combines the LVM + > mkfs steps above) but I don't know of any clusters using ZFS to manage > local file systems on the compute nodes :-) > > > One /could/ leverage XFS project quotas. E.g. for Slurm job 2147483647: > > > *[root@r00n00 /]# mkdir /tmp-alloc/slurm-2147483647* > *[root@r00n00 /]# xfs_quota -x -c 'project -s -p > /tmp-alloc/slurm-2147483647 2147483647' /tmp-alloc* > Setting up project 2147483647 (path /tmp-alloc/slurm-2147483647)... > Processed 1 (/etc/projects and cmdline) paths for project > 2147483647 with recursion depth infinite (-1). > *[root@r00n00 /]# xfs_quota -x -c 'limit -p bhard=1g 2147483647' > /tmp-alloc* > *[root@r00n00 /]# cd /tmp-alloc/slurm-2147483647* > *[root@r00n00 slurm-2147483647]# dd if=/dev/zero of=zeroes bs=5M > count=1000* > dd: error writing ‘zeroes’: No space left on device > 205+0 records in > 204+0 records out > 1073741824 bytes (1.1 GB) copied, 2.92232 s, 367 MB/s > > : > > [root@r00n00 /]# rm -rf /tmp-alloc/*slurm-2147483647* > [root@r00n00 /]# *xfs_quota -x -c 'limit -p bhard=0 2147483647' > /tmp-alloc* > > > Since Slurm jobids max out at 0x03FFFFFF (and 2147483647 = 0x7FFFFFFF) > we have an easy on-demand project id to use on the file system. Slurm > tmpfs plugins have to do a mkdir to create the per-job directory, > adding two xfs_quota commands (which run in more or less O(1) time) > won't extend the prolog by much. Likewise, Slurm tmpfs plugins have to > scrub the directory at job cleanup, so adding another xfs_quota > command will not do much to change their epilog execution times. The > main question is "where does the tmpfs plugin find the quota limit for > the job?" > > > > > >> On Feb 6, 2024, at 08:39, Tim Schneider via slurm-users >> <slurm-users(a)lists.schedmd.com> wrote: >> >> Hi, >> >> In our SLURM cluster, we are using the job_container/tmpfs plugin to >> ensure that each user can use /tmp and it gets cleaned up after them. >> Currently, we are mapping /tmp into the nodes RAM, which means that >> the cgroups make sure that users can only use a certain amount of >> storage inside /tmp. >> >> Now we would like to use of the node's local SSD instead of its RAM >> to hold the files in /tmp. I have seen people define local storage as >> GRES, but I am wondering how to make sure that users do not exceed >> the storage space they requested in a job. Does anyone have an idea >> how to configure local storage as a proper tracked resource? >> >> Thanks a lot in advance! >> >> Best, >> >> Tim >> >> >> -- >> slurm-users mailing list -- slurm-users(a)lists.schedmd.com >> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com >

2 1

scheme for protected GPU jobs from preemption
by Paul Raines 06 Feb '24

06 Feb '24

After using just Fairshare for over a year on our GPU cluster, we have decided it is not working for us for what we really want to achieve among our groups. We have decided to look at preemption. What we want is for users to NOT have a #job/GPU maximum (if they are only person on the cluster they should be able to use it all), but if another user comes to the "full" cluster they should immediately be able to run some jobs. Thus preemption is needed. In our scheme we want * users to have N protected GPU jobs that cannot be preempted where N is the number of GPUs allocated. * N may not be the same for all users. Some priviledged users get more. * jobs pending in the queue will have lower priority dependent on the number of GPUs allocated to running jobs. Maybe doable somehow with PriorityWeightJobSize though not sure how. * Jobs over N are subject to preemption (and requeued if --requeue is given) with shortest running jobs of the user with most unprotected GPUs preempted first. * another complication is we have a variety of different GPUs and users may ask for specific ones which can limit what unprotected GPU jobs are available for preemption My first attempt to do this in SLURM was to just create two partitions, GPU and GPU-req, with different PriorityTier values and the later partition have PreemptMode=REQUEUE. But N would be set by a MaxTRES on the first partition and be the same for everyone and we need it to be INDEPENDENT for each user. Also users would have to "think" about which partition to submit jobs to. And users want their longest running "unprotected" job to be able to be PROMOTED to a "protected" jobs automatically when a "protected" job finishes. However slurm does not allow running jobs to move between partitions. I am trying to figure out QOS pre-emption which might solve the independent N per user issue but I don't think it will solve the promotion issue. Any ideas how this scheme might be possible in SLURM? Otherwise I might have to write a complicated cron job that tries to do it all "outside" of SLURM issuing scontrol commands. --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

Restricting local disk storage of jobs
by Tim Schneider 06 Feb '24

06 Feb '24

Hi, In our SLURM cluster, we are using the job_container/tmpfs plugin to ensure that each user can use /tmp and it gets cleaned up after them. Currently, we are mapping /tmp into the nodes RAM, which means that the cgroups make sure that users can only use a certain amount of storage inside /tmp. Now we would like to use of the node's local SSD instead of its RAM to hold the files in /tmp. I have seen people define local storage as GRES, but I am wondering how to make sure that users do not exceed the storage space they requested in a job. Does anyone have an idea how to configure local storage as a proper tracked resource? Thanks a lot in advance! Best, Tim

3 2

Re: [ext] Restricting local disk storage of jobs
by Tim Schneider 06 Feb '24

06 Feb '24

Hi Magnus, I understand. Thanks a lot for your suggestion. Best, Tim On 06.02.24 15:34, Hagdorn, Magnus Karl Moritz wrote: > Hi Tim, > in the end the InitScript didn't contain anything useful because > > slurmd: error: _parse_next_key: Parsing error at unrecognized key: > InitScript > > At this stage I gave up. This was with SLURM 23.02. My plan was to > setup the local scratch directory with XFS and then get the script to > apply a project quota, ie quota attached to the directory. > > I would start by checking if slurm recognises the InitScript option. > > Regards > magnus > > On Tue, 2024-02-06 at 15:24 +0100, Tim Schneider wrote: >> Hi Magnus, >> >> thanks for your reply! If you can, would you mind sharing the >> InitScript >> of your attempt at getting it to work? >> >> Best, >> >> Tim >> >> On 06.02.24 15:19, Hagdorn, Magnus Karl Moritz wrote: >>> Hi Tim, >>> we are using the container/tmpfs plugin to map /tmp to a local NVMe >>> drive which works great. I did consider setting up directory >>> quotas. I >>> thought the InitScript [1] option should do the trick. Alas, I >>> didn't >>> get it to work. If I remember correctly, slurm complained about the >>> option being present. In the end we recommend our users to make >>> exclusive use a node if they are going to use a lot of local >>> scratch >>> space. I don't think this happens very often if at all. >>> Regards >>> magnus >>> >>> [1] >>> https://slurm.schedmd.com/job_container.conf.html#OPT_InitScript >>> >>> >>> On Tue, 2024-02-06 at 14:39 +0100, Tim Schneider via slurm-users >>> wrote: >>>> Hi, >>>> >>>> In our SLURM cluster, we are using the job_container/tmpfs plugin >>>> to >>>> ensure that each user can use /tmp and it gets cleaned up after >>>> them. >>>> Currently, we are mapping /tmp into the nodes RAM, which means >>>> that >>>> the >>>> cgroups make sure that users can only use a certain amount of >>>> storage >>>> inside /tmp. >>>> >>>> Now we would like to use of the node's local SSD instead of its >>>> RAM >>>> to >>>> hold the files in /tmp. I have seen people define local storage >>>> as >>>> GRES, >>>> but I am wondering how to make sure that users do not exceed the >>>> storage >>>> space they requested in a job. Does anyone have an idea how to >>>> configure >>>> local storage as a proper tracked resource? >>>> >>>> Thanks a lot in advance! >>>> >>>> Best, >>>> >>>> Tim >>>> >>>>

1 0

Re: [ext] Restricting local disk storage of jobs
by Tim Schneider 06 Feb '24

06 Feb '24

Hi Magnus, thanks for your reply! If you can, would you mind sharing the InitScript of your attempt at getting it to work? Best, Tim On 06.02.24 15:19, Hagdorn, Magnus Karl Moritz wrote: > Hi Tim, > we are using the container/tmpfs plugin to map /tmp to a local NVMe > drive which works great. I did consider setting up directory quotas. I > thought the InitScript [1] option should do the trick. Alas, I didn't > get it to work. If I remember correctly, slurm complained about the > option being present. In the end we recommend our users to make > exclusive use a node if they are going to use a lot of local scratch > space. I don't think this happens very often if at all. > Regards > magnus > > [1] > https://slurm.schedmd.com/job_container.conf.html#OPT_InitScript > > > On Tue, 2024-02-06 at 14:39 +0100, Tim Schneider via slurm-users wrote: >> Hi, >> >> In our SLURM cluster, we are using the job_container/tmpfs plugin to >> ensure that each user can use /tmp and it gets cleaned up after them. >> Currently, we are mapping /tmp into the nodes RAM, which means that >> the >> cgroups make sure that users can only use a certain amount of storage >> inside /tmp. >> >> Now we would like to use of the node's local SSD instead of its RAM >> to >> hold the files in /tmp. I have seen people define local storage as >> GRES, >> but I am wondering how to make sure that users do not exceed the >> storage >> space they requested in a job. Does anyone have an idea how to >> configure >> local storage as a proper tracked resource? >> >> Thanks a lot in advance! >> >> Best, >> >> Tim >> >>

2 1

Starting a job after a file is created in previous job (dependency looking for soluton)
by Amjad Syed 06 Feb '24

06 Feb '24

Hello I have the following scenario: I need to submit a sequence of up to 400 jobs where the even jobs depend on the preceeding odd job to finish and every odd job depends on the presence of a file generated by the preceding even job (availability of the file for the first of those 400 jobs is guaranteed). If I just submit all those jobs via a loop using dependencies, then I end up with a lot of pending jobs who might later not even run because no output file has been produced by the preceding jobs. Is there a way to pause the submission loop until the required file has been generated so that at most two jobs are submitted at the same time? Here is a sample submission script showing what I want to achieve. for i in {1..200}; do FILE=GHM_paramset_${i}.dat # How can I pause the submission loop until the FILE has been created???? #if test -f "$FILE"; then jobid4=$(sbatch --parsable --dependency=afterok:$jobid3 job4_sub $i) jobid3=$(sbatch --parsable --dependency=afterok:$jobid4 job3_sub $i) #fi done Any help will be appreciated Amjad

3 2

SLURM configuration for LDAP users
by Richard Chang 05 Feb '24

05 Feb '24

Hi, I am a little new to this, so please pardon my ignorance. I have configured slurm in my cluster and it works fine with local users. But I am not able to get it working with LDAP/SSSD authentication. User logins using ssh are working fine. An LDAP user can login to the login, slurmctld and compute nodes, but when they try to submit jobs, slurmctld logs an error about invalid account or partition for user. Someone said we need to add the user manually into the database using the sacctmgr command. But I am not sure we need to do this for each and every LDAP user. Yes, it does work if we add the LDAP user manually using sacctmgr. But I am not convinced this manual way is the way to do. The documentation is not very clear about using LDAP accounts. Saw somewhere in the list about using UsePAM=1 and copying or creating a softlink for slurm PAM module under /etc/pam.d . But it didn't work for me. Saw somewhere else that we need to specifying LaunchParameters=enable_nss_slurm in the slurm.conf file and put slurm keyword in passwd/group entry in the /etc/nsswitch.conf file. Did these, but didn't help either. I am bereft of ideas at present. If anyone has real world experience and can advise, I will be grateful. Thank you, Richard

4 4

timeout of get_user_env does not obey time limit
by urbanjost 03 Feb '24

03 Feb '24

If I use the sbatch(1) option --export=NONE or wipe the environment with "env -i /usr/bin/sbatch ..." or use --export=NIL then the environment is not properly constructed and I see the message in the /var/log/*slurm* files: [2024-02-03T11:50:33.052] _get_user_env: get env for user jsu here [2024-02-03T11:52:33.152] timeout waiting for /bin/su to complete [2024-02-03T11:52:34.152] error: Failed to load current user environment variables [2024-02-03T11:52:34.153] error: _get_user_env: Unable to get user's local environment, running only with passed environment This occurs at 120 seconds not matter if I add --get-user-env=3600 or adjust many slurm.conf time-related parameters. It is easy to reproduce by adding "sleep 100" into a .cshrc file and sbatch(1) the file #!/bin/csh #SBATCH --export=NONE --propagate=NONE --get-user-env=3600L printenv HOME printenv USER printenv PATH env I have adjust MANY time-related limits in the slurm.conf file to no avail. When the system is unresponsive or heavily loaded or users have prologues that set up complex environments via module commands (which can be notoriously slow) the jobs are failing or producing errors. If I configure Slurm so that jobs that timeout requeue instead of running then a user with a slow login setup can submit a large number of jobs and basically close down a cluster because this option not only requeues jobs that fail but puts the node it occurred on in a DRAIN state. We see this as very dangerous as by defaults jobs proceed to execute even when their environment is not properly constructed. I can see that "slurmrestd getenv" and the procedure get_user_env(3c) are involved, but a preliminary scan of the code looked like the --get-user-env=NNNN value was being parsed, and I did not see a reason the setup always times out at 120 seconds (at least on my system). Does anyone know how to get the time allowed to get the default user environment to use the value on the --get-user-env option when no environment is being exported to a job? This is showing up sporadically and causing intermittent failures that are very confusing and disturbing to the users it occurs with. Sent with [Proton Mail](https://proton.me/) secure email.

1 0

selecting job-specific log messages
by urbanjost 03 Feb '24

03 Feb '24

log files use many strings to identify job, including not having a jobID in the message NUMBER=$SLURM_JOBID egrep "\.\<$NUMBER\>\] |\<$NUMBER\>\.batch|jobid \<$NUMBER\>|JObId=\<$NUMBER\>|job id \<$NUMBER\>|job\.\<$NUMBER\>|job \<$NUMBER\>|jobid \[\<$NUMBER\>\]|task_p_slurmd_batch_request: \<$NUMBER\>" /var/log/slurm* Even that misses cruciall data that does not even contain the jobid [2024-02-03T11:50:33.052] _get_user_env: get env for user jsu here [2024-02-03T11:52:33.152] timeout waiting for /bin/su to complete [2024-02-03T11:52:34.152] error: Failed to load current user environment variables [2024-02-03T11:52:34.153] error: _get_user_env: Unable to get user's local environment, running only with passed environment It would be very useful if all messages related to a job had a consistent string in them for grepping the log files; even better might be a command like "scontrol show jobid=NNNN log_messages But I could not find what I wanted (an easy way to find all daemon log messages related to a specific job). I would find it particularly useful if there were a way to automatically append such information to the stdout of the job at job termination so users would automatically get information about job failures or warnings. Is there such a feature available I have missed? Sent with [Proton Mail](https://proton.me/) secure email.

1 0

after upgrade to 23.11.1 nodes stuck in completion state
by Paul Raines 01 Feb '24

01 Feb '24

I finally had downtime on our cluster running 20.11.3 and decided to upgrade SLURM. All daemons were stopped on nodes and master. Rocky 8 Linux OS was updated but not changed configuration-wise in anyway. On the master, when I first installed 23.11.1 and tried to run slurmdbd -D -vvv at the command line, it balked as the more than 2 major version jump. I then installed 22.05.6 and run slurmdbd -D -vvv which took a long time but completed fine. I reinstalled 23.11.1 and did slurmdbd -D --vvv which went quick. I stopped it then ran slurmdbd normally with systemctl. When I ran slurmctld it complained about incompatable state files. I didn't care about loosing state so ran with the -i option to ignore. Seemed happy. I then started up slurmd on my various GPU and non-GPU nodes. I ran a test interactive job just to see if I would get a shell with expected SLURM environment on the nodes and it worked. Except on the GPU nodes I got errors about not being able to get/set GPU frequences. I removed GpuFreqDef=medium from slurm.conf and that went away. I had my people start running jobs last night. This morning I come in and see a lot of nodes are stuck in completing state. All these are GPU nodes. For example on node 'sevilla' squeue reports JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 3679433 lcna100 sjob_123 sg1526 CG 8:29 1 sevilla 3679432 lcna100 sjob_123 sg1526 CG 8:35 1 sevilla 3679431 lcna100 sjob_123 sg1526 CG 8:42 1 sevilla In the slurmd.log on sevilla I see every minute the lines: [2024-01-28T09:37:52.003] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug: _rpc_terminate_job: uid = 1150 JobId=3679432 [2024-01-28T09:37:52.004] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug: _rpc_terminate_job: uid = 1150 JobId=3679431 [2024-01-28T09:37:52.004] debug2: Finish processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug: _rpc_terminate_job: uid = 1150 JobId=3679433 [2024-01-28T09:37:52.004] debug2: Finish processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Finish processing RPC: REQUEST_TERMINATE_JOB Back at the time 3679433 was cancelled by the user I see [2024-01-28T00:27:48.658] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T00:27:48.658] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T00:27:48.658] debug: _rpc_terminate_job: uid = 1150 JobId=3679433 [2024-01-28T00:27:48.658] debug: credential for job 3679433 revoked [2024-01-28T00:27:48.658] debug: _rpc_terminate_job: sent SUCCESS for 3679433, waiting for prolog to finish [2024-01-28T00:27:48.658] debug: Waiting for job 3679433's prolog to complete On sevilla itself there are no processes currently running for user sg1526 so no sign of these jobs. There ARE processes running for user mu40 though [root@sevilla ~]# pgrep -f slurm_script | xargs -n1 pstree -p | grep ^slurm slurm_script(30691)---starter-suid(30764)-+-python3(30784)-+-{python3}(30817) slurm_script(30861)---starter-suid(30873)-+-python3(30888)-+-{python3}(30915) slurm_script(30963)---starter-suid(30975)-+-python3(30992)-+-{python3}(31020) [root@sevilla ~]# strings /proc/30691/cmdline /bin/sh /var/slurm/spool/d/job3679220/slurm_script /usr/bin/sacct -p --jobs=3679220 --format=JobID,User,ReqTRES,NodeList,Start,End,Elapsed,CPUTime,State,ExitCode JobID|User|ReqTRES|NodeList|Start|End|Elapsed|CPUTime|State|ExitCode| 3679220|mu40|billing=6,cpu=3,gres/gpu=1,mem=32G,node=1|sevilla|2024-01-27T22:33:27|2024-01-27T23:26:39|00:53:12|02:39:36|NODE_FAIL|1:0| 3679220.batch|||sevilla|2024-01-27T22:33:27|2024-01-27T23:26:39|00:53:12|02:39:36|CANCELLED|| 3679220.extern|||sevilla|2024-01-27T22:33:27|2024-01-27T23:26:39|00:53:12|02:39:36|CANCELLED|| The slurmd.log has at the time it started lines like: [2024-01-27T22:33:28.679] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30691, probably hasn't started yet or has already finished [2024-01-27T22:33:58.092] [3679220.extern] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30650, probably hasn't started yet or has already finished [2024-01-27T22:33:58.166] [3679220.batch] debug2: profile signaling type Task [2024-01-27T22:33:58.168] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30691, probably hasn't started yet or has already finished [2024-01-27T22:33:58.170] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30764, probably hasn't started yet or has already finished [2024-01-27T22:33:58.172] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30784, probably hasn't started yet or has already finished ... [2024-01-27T22:46:39.018] debug: _step_connect: connect() failed for /var/slurm/spool/d/sevilla_3679220.4294967292: Connection refused [2024-01-27T22:46:39.018] debug: Cleaned up stray socket /var/slurm/spool/d/sevilla_3679220.4294967292 [2024-01-27T22:46:39.018] debug: _step_connect: connect() failed for /var/slurm/spool/d/sevilla_3679220.4294967291: Connection refused [2024-01-27T22:46:39.018] debug: Cleaned up stray socket /var/slurm/spool/d/sevilla_3679220.4294967291 [2024-01-27T22:46:39.018] _handle_stray_script: Purging vestigial job script /var/slurm/spool/d/job3679220/slurm_script then later at the time of termination [2024-01-27T23:26:39.042] debug: _rpc_terminate_job: uid = 1150 JobId=3679220 [2024-01-27T23:26:39.042] debug: credential for job 3679220 revoked [2024-01-27T23:26:39.042] debug2: No steps in jobid 3679220 to send signal 998 [2024-01-27T23:26:39.043] debug2: No steps in jobid 3679220 to send signal 18 [2024-01-27T23:26:39.043] debug2: No steps in jobid 3679220 to send signal 15 [2024-01-27T23:26:39.044] debug2: set revoke expiration for jobid 3679220 to 1706416119 UTS [2024-01-27T23:26:39.044] debug: Waiting for job 3679220's prolog to complete [2024-01-27T23:26:39.044] debug: Finished wait for job 3679220's prolog to complete [2024-01-27T23:26:39.068] debug: completed epilog for jobid 3679220 [2024-01-27T23:26:39.071] debug: JobId=3679220: sent epilog complete msg: rc = 0 So I have a node stuck in 'comp' state that claims to have jobs completing that are definite NOT running on the box BUT there are jobs running on the box that SLURM thinks are done --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

4 7

Why is Slurm 20 the latest RPM in RHEL 8/Fedora repo?
by Robert Kudyba 31 Jan '24

31 Jan '24

According to these links: https://rpmfind.net/linux/rpm2html/search.php?query=slurm https://src.fedoraproject.org/rpms/slurm Why doesn't RHEL 8 get a newer version? Can someone update the repo maintainer Philip Kovacs <https://src.fedoraproject.org/user/pkfed> < pkfed(a)fedoraproject.org>? There was a ticket at https://bugzilla.redhat.com/show_bug.cgi?id=1912491 but no movement on RHEL 8.

5 4

Mailing list upgrade - slurm-users list paused
by Tim Wickberg 30 Jan '24

30 Jan '24

Hey folks - The mailing list will be offline for about an hour as we upgrade the host, upgrade the mailing list software, and change the mail configuration around. As part of these changes, the "From: " field will no longer be the original sender, but instead use the mailing list ID itself. This is to comply with DMARC sending options, and allow us to start DKIM signing messages to ensure deliverability once Google and Yahoo impose new policy changes in February. This is the last post on the current (mailman2) list. I'll send a welcome message on the upgraded (mailman3) list once finished, and when the list is open to new traffic again. - Tim -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 1

Testing to SchedMD folks
by Tim Wickberg 30 Jan '24

30 Jan '24

Hey - this is hopefully a final test before the list is re-enabled for everyone. - Tim

1 0

Socket timed out - tuning
by Reed Dier 29 Jan '24

29 Jan '24

Hoping someone can help point me towards some tweaks to help prevent denial-of-service issues. > sbatch: error: Batch job submission failed: Socket timed out on send/recv operation Root cause is understood, issues with shared storage for the slurmctld’s was impacted, leading to an increase in write latency to the StateSaveLocation. Then with a large enough avalanche of job submissions, things the RPC’s would stack up and stop responding. I’ve been running well with some tweaks sourced from the “high-throughput” guide <https://slurm.schedmd.com/high_throughput.html>. > SchedulerParameters=max_rpc_cnt=400,\ > sched_min_interval=50000,\ > sched_max_job_start=300,\ > batch_sched_delay=6 > KillWait=30 > MessageTimeout=30 I’m assuming that I was running into batch_sched_delay because looking at sdiag after the fact, it was averaging .2s, and total time is 5.5h out of 16h8m18s at the time of the sdiag sample. > ******************************************************* > sdiag output at Thu Jan 25 11:08:18 2024 (1706198898) > Data since Wed Jan 24 19:00:00 2024 (1706140800) > ******************************************************* > REQUEST_SUBMIT_BATCH_JOB ( 4003) count:98400 ave_time:201442 total_time:19821991013 Currently on 22.05.8, but hoping to get to 23.02.7 soon™, and I think this could possible resolve the issue well enough if I’m reading it correctly from the release notes <https://slurm.schedmd.com/archive/slurm-23.02-latest/news.html>? > HIGHLIGHTS > ========== > -- slurmctld - Add new RPC rate limiting feature. This is enabled through > SlurmctldParameters=rl_enable, otherwise disabled by default. > rl_enable <https://slurm.schedmd.com/slurm.conf.html#OPT_rl_enable>Enable per-user RPC rate-limiting support. Client-commands will be told to back off and sleep for a second once the limit has been reached. This is implemented as a "token bucket", which permits a certain degree of "bursty" RPC load from an individual user before holding them to a steady-state RPC load established by the refill period and rate. But given that the hardware seems to be well over provisioned, CPU never drops below 5% idle, it feels like there is more room to squeeze some optimization out of here that I’m missing in the interim, and hoping to get a better overall understanding in the process. I scrape the DBD Agent queue size from sdiag every 30s and the largest value I saw was 115, which is much higher than normal, but should be well below MaxDBDMsgs, where the minimum value is 10000. I would really hope that I didn’t potentially hit a 30s MessageTimeout value, but I guess thats on the table all well, not knowing if that would potentially trigger the sbatch submission failure like that. Just moving the max_rpc_cnt value up seems like an easy button, but also seems like it could have some adverse effects for backfill scheduling, and may be diminishing returns for actually keeping RPCs flowing? > Setting max_rpc_cnt to more than 256 will be only useful to let backfill continue scheduling work after locks have been yielded (i.e. each 2 seconds) if there are a maximum of MAX(max_rpc_cnt/10, 20) RPCs in the queue. i.e. max_rpc_cnt=1000, the scheduler will be allowed to continue after yielding locks only when there are less than or equal to 100 pending RPCs. Obviously, fix the storage is the real solution, but hoping that there may be more goodness to unlock, even if it is as simple as “upgrade to 23.02”. Appreciate any insight, Reed

1 0

Two jobs each with a different partition running on same node?
by Loris Bennett 29 Jan '24

29 Jan '24

Hi, I seem to remember that in the past, if a node was configured to be in two partitions, the actual partition of the node was determined by the partition associated with the jobs running on it. Moreover, at any instance where the node was running one or more jobs, the node could only actually be in a single partition. Was this indeed the case and is it still the case with version Slurm 23.02.7? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT (ex-ZEDAT), Freie Universität Berlin

2 1

2025

2024

slurm-users ----- 2025 ----- April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users