- slurm-users - lists.schedmd.com

Partition, Qos Limits & Scheduling of large jobs
by Muck, Katrin 28 Feb '24

28 Feb '24

Hi everyone! I read the slurm documentation about qos, resource limits, scheduling and priority now multiple times and even looked into the slurm source but I'm still not sure if I got everything correctly, so this is why I decided to ask here ... The problem: we see the effect that sometimes larger jobs with e.g. 16 gpus in our (small) gpu queue get delayed and shifted back without any reason that is apparent to us. small jobs that only use e.g. 1 or 2 gpus get scheduled much quicker even though they have a runtime of 3 days ... What we want to do: - We have a number of nodes with 2x gpus that are usable by the users of our cluster - Some of these nodes belong to so called 'private projects'. Private projects have higher priority than other projects. Attached to that is a contingent of nodes & "guaranteed" nodes e.g. they could have a contingent of 4 nodes (8 gpus) and e.g. 2 "guaranteed" nodes (4 gpus) - Guaranteed nodes are nodes that should always be kept idle for the private project, so users of the private project can immediately schedule work on those nodes - The other nodes are shared with other projects in general if they are not "in use" How we are currently doing this (it has history): Lets assume we have 50 nodes and 100 gpus. - We have a single partition for all gpu nodes (e.g. 50 nodes) - Private projects have private queues with a very high priority and a gres limit of the number of gpus they reserved (e.g. 10 nodes -> 20 gpus) - Normal projects only have access to the public queue and schedule work there. - This public queue has an upper gres limit of "total number of gpus" - "guaranteed gpus of all private projects" (e.g. 50 - 10 nodes -> 40 nodes -> 80 gpus). Regarding the scheduler, we currently use the following settings: SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_CORE_MEMORY,CR_CORE_DEFAULT_DIST_BLOCK,CR_ONE_TASK_PER_CORE SchedulerParameters=defer,max_sched_time=4,default_queue_depth=1000,partition_job_depth=500,enable_user_top,bf_max_job_user=20,bf_interval=120,bf_window=4320,bf_resolution=1800,bf_continue Partition/queue depth is deliberately set high at the moment to avoid problems with jobs not even being examined. The problem in more detail: One of the last jobs (16 gpus needed) we diagnosed had a approximate start time that was beyond all end times of running/scheduled jobs - like jobs on feb 22 would release more than enough gpus so the job could have been immediately scheduled afterwards, but the start time was still feb 23. Priority wise the job had the highest priority of all pending jobs for the partition. When we turned on scheduler debugging and increased log levels, we observed the following messages for this job: JobId=xxxx being held, if allowed the job request will exceed QOS xxxxx group max tres(gres/gpu) limit yy with already used yy + requested 16 followed by sched: JobId=2796696 delayed for accounting policy So to us this meant that the scheduler was always hitting the qos limits, which makes sense because the usage is always very high in the gpu queue and thus the job wasn't scheduled ... At first we were worried, that this meant that "held"/"delayed" jobs like this would never actually get scheduled when contention is high enough e.g. small jobs getting backfilled in and thus qos limits stay at max for a long time. But for some reason we could not determine the job eventually got scheduled at one point and then ran at the scheduled start time. Open Questions: - why it couldn't be scheduled in the first place. initially we thought (from the source code i looked into) the "delayed for accounting policy" prevents further scheduling in general, but since it was scheduled this assumption must be wrong? - why it was scheduled at some point. when it was scheduled, contention was still high and the qos limits definitely still applied - how we could modify the current setup so that the scheduling of larger jobs becomes "better" and more reproducible/explainable Apart from all of this I'm also asking myself if there is maybe a better way to setup a system that works the way we want? This got a bit long but I hope its clear enough :) Kind regards, Katrin

1 0

"Optimal" slurm configuration
by Max Grönke 26 Feb '24

26 Feb '24

Hello! In our current cluster the workflows are quite diverse (bunch of large, long (24-72h) jobs; medium size <4h job; and many small 1 node jobs). The current priority is fair share only (averaged on a ~few months timescale). For the new setup we would like to (1) discourage the 1 node jobs [especially the long ones] (2) have a fair use of the machine for the different cases. In particular, we found that the waiting time / runtime ratio is very large for the ~1-4h jobs. For this reason, I have been trying to read up on strategies in terms of partition / priority politics but could not find any good documentation in this regard (as https://slurm.schedmd.com/documentation.html is more focused on the technical aspects). Does anyone know any good references for this? In particular, we were thinking about (a) somehow have a long-term as well as shorter term fairshare count, i.e., lowering the priority if a user has something running already or (b) introduce a "small" partition for the <4h jobs with higher priority but we're unsure if this will block all the larger jobs to run... Any help or reading material is highly appreciated. Many thanks, Max

2 1

FAQ describing how to hold a job ignores scontrol subcommands specifically for that purpose
by urbanjost 25 Feb '24

25 Feb '24

There are scontrol subcommands uhold/hold/release/requeuehold that are ignored when describing how to place a job on hold in FAQ 21; and it is never explained why the method described therein is the best method, it just states it is. Does anyone know why the FAQ method is better than using the subcommands? Is it because the PRIORITY and/or NICE values are not altered (maybe)? The question is also about Running but the answer is just about Starting and not Suspending which is not quite as clear (I think "running" should be "starting" to make that clear; and/or how to suspend should be described as well). If the answer is not clear to anyone, I might turn this into a request for clarification in the Slurm bugzilla as a documentation change request but wanted to see if this was already clear to anyone and I am missing something. From FAQ: 21. How can I temporarily prevent a job from running (e.g. place it into a hold state)? The easiest way to do this is to change a job's earliest begin time (optionally set at job submit time using the --begin option). The example below places a job into hold state (preventing its initiation for 30 days) and later permitting it to start now. <METHOD I> $ scontrol update JobId=1234 StartTime=now+30days ... later ... $ scontrol update JobId=1234 StartTime=now Note: Empirically in METHOD I the JobId can be a <job_list> , which I initially thought required single JobIDs. No explanation is given on why METHOD I is best; and there are other methods that seem more intuitive. I wonder what is undesirable about the following method which I have been using -- using the scontrol(1) subcommands hold/uhold/release/requeuehold. <METHOD II> $ scontrol hold <job_list> # advantage to administrator as user cannot change $ scontrol uhold <job_list> $ scontrol release <job_list> Examples: $ scontrol uhold jobname=JOB_NAME $ scontrol uhold '[100-200],300,500' Using uhold the "Reason" changes to something easily identifying the job is being held, as "Reason=None" became "Reason=JobHeldUser which seems better that Method I in that regard. The downside might be PRIORITY changed to zero and then went to a very large value when released? Another method appears to be that setting PRIORITY to zero also places jobs in hold. <METHOD III> $ scontrol update jobid=373 Priority=0 $ scontrol release jobid=373 # sets to a very high value $ scontrol update jobid=373 Priority=11111 # put back to lower desired value Once lowered, does an optional setting prevent a user from raising PRIORITY(?) The manual says Only the Slurm administrator or root can increase job's priority. At least on my machine the "release" buts the priority to a very high value, and a regular user can lower the value back to the (probably) lower original value. I did not see it happening but there are some statements in the documentation that make me think not only PRIORITY but perhaps the NICE value might be changed by METHOD II and METHOD III, although I could not get the NICE value to be inadvertently changed. Sent with [Proton Mail](https://proton.me/) secure email.

1 1

slurm-23.11.3-1 with X11 and zram causing permission errors: error: _forkexec_slurmstepd: slurmstepd failed to send return code got 0: Resource temporarily unavailable; Requeue of JobId=530 returned an error: Only batch jobs are accepted or processed
by Robert Kudyba 24 Feb '24

24 Feb '24

We switched over from using systemctl for tmp.mount and change to zram, e.g., modprobe zram echo 20GB > /sys/block/zram0/disksize mkfs.xfs /dev/zram0 mount -o discard /dev/zram0 /tmp srun with --x11 was working before changing this. We're on RHEL 9. slurmctld logs show this whenever --x11 is used with srun: [2024-02-23T20:22:43.442] [529.extern] error: setup_x11_forward: failed to create temporary XAUTHORITY file: Permission denied [2024-02-23T20:22:43.442] [529.extern] error: x11 port forwarding setup failed [2024-02-23T20:22:43.442] error: _forkexec_slurmstepd: slurmstepd failed to send return code got 0: Resource temporarily unavailable [2024-02-23T20:22:43.443] Could not launch job 529 and not able to requeue it, cancelling job [2024-02-23T20:26:15.881] [530.extern] error: setup_x11_forward: failed to create temporary XAUTHORITY file: Permission denied [2024-02-23T20:26:15.881] [530.extern] error: x11 port forwarding setup failed [2024-02-23T20:26:15.882] error: _forkexec_slurmstepd: slurmstepd failed to send return code got 0: Resource temporarily unavailable [2024-02-23T20:26:15.883] Could not launch job 530 and not able to requeue it, cancelling job slurmd log entries from a node: [2024-02-23T20:26:15.859] sched: _slurm_rpc_allocate_resources JobId=530 NodeList=2402-node005 usec=1800 [2024-02-23T20:26:15.882] _slurm_rpc_requeue: Requeue of JobId=530 returned an error: Only batch jobs are accepted or processed [2024-02-23T20:26:15.883] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=530 uid 0 [2024-02-23T20:26:15.962] _slurm_rpc_complete_job_allocation: JobId=530 error Job/step already completing or completed srun -v --pty -t 0-4:00 --x11 --mem=10g srun: defined options srun: -------------------- -------------------- srun: account : me srun: mem : 10G srun: nodelist : our-node srun: pty : srun: time : 04:00:00 srun: verbose : 1 srun: x11 : all srun: -------------------- -------------------- srun: end of defined options srun: Waiting for resource configuration srun: error: Nodes our-node are still not ready srun: error: Something is wrong with the boot of the nodes. slurm.conf has PrologFlags=x11 set. /usr/bin/xauth is installed on each compute node. Is this a known issue with zram or is that just a red herring and there's something else wrong?

3 5

Slurm version 23.11.4 is now available
by Tim McMullan 22 Feb '24

22 Feb '24

We are pleased to announce the availability of Slurm version 23.11.4. The 23.11.4 release includes a number of fixes to stability and various bug fixes. Some notable changes include that VSZ is no longer being reported when using cgroup/v2 (this is not provided by the kernel), a warning has been added if using select/linear and tolology/tree together as this will not be supported in the next major release, and a backwards compatibility issue that caused jobs using --gpus to be rejected when submitted from 23.02 or 22.05. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.4 > ========================== > -- Fix a memory leak when updating partition nodes. > -- Don't leave a partition around if it fails to create with scontrol. > -- Fix segfault when creating partition with bad node list from scontrol. > -- Fix preserving partition nodes on bad node list update from scontrol. > -- Fix assertion in developer mode on a failed message unpack. > -- Fix repeat POWER_DOWN requests making the nodes available for ping. > -- Fix rebuilding job alias_list on restart when nodes are still powering up. > -- Fix INVALID nodes running health check. > -- Fix cloud/future nodes not setting addresses on invalid registration. > -- scrun - Remove the requirement to set the SCRUN_WORKING_DIR environment > variable. This was a regression in 23.11. > -- Add warning for using select/linear with topology/tree. > This combination will not be supported in the next major version. > -- Fix health check program not being run after first pass of all nodes when > using MaxNodeCount. > -- sacct - Set process exit code to one for all errors. > -- Add SlurmctldParameters=disable_triggers option. > -- Fix issue running steps when the allocation requested an exclusive > allocation shards along with shards. > -- Fix cleaning up the sleep process and the cgroup of the extern step if > slurm_spank_task_post_fork returns an error. > -- slurm_completion - Add missing --gres-flags= options > multiple-tasks-per-sharing and one-task-per-sharing. > -- scrun - Avoid race condition that could cause outbound network > communications to incorrectly rejected with an incomplete packet error. > -- scrun - Gracefully handle kernel giving invalid expected number of incoming > bytes for a connection causing incoming packet corruption resulting in > connection getting closed. > -- srun - return 1 when a step lauch fails > -- scrun - Avoid race condition that could cause deadlock during shutdown. > -- Fix scontrol listpids to work under dynamic node scenarios. > -- Add --tres-bind to --help and --usage output. > -- Add --gres-flags=allow-task-sharing to allow GPUs to still be accessible > among all tasks when binding GPUs to specific tasks. > -- Fix issue with CUDA_VISIBLE_DEVICES showing the same MIG device for all > tasks when using MIGs with --tres-per-task or --gpus-per-task. > -- slurmctld - Prevent a potential hang during shutdown/reconfigure if the > association cache thread was previously shut down. > -- scrun - Avoid race condition that could cause scrun to hang during > shutdown when connections have pending events. > -- scrun - Avoid excessive polling of connections during shutdown that could > needlessly cause 100% CPU usage on a thread. > -- sbcast - Use user identity from broadcast credential instead of looking it > up locally on the node. > -- scontrol - Remove "abort" option handling. > -- Fix an error message referring to the wrong RPC. > -- Fix memory leak on error when creating dynamic nodes. > -- Fix a slurmctld segfault when a cloud/dynamic node changes hostname on > registration. > -- Prevent a slurmctld deadlock if the gpu plugin fails to load when > creating a node. > -- Change a slurmctld fatal() to an error() when attempting to create a > dynamic node with a global autodetect set in gres.conf. > -- Fix leaving node records on error when creating nodes with scontrol. > -- scrun/sackd - Avoid race condition where shutdown could deadlock. > -- Fix a regression in 23.02.5 that caused pam_slurm_adopt to fail when > the user has multiple jobs on a node. > -- Add GLOB_SILENCE flag that silences the error message which will display if > an include directive attempts to use the "*" wildcard. > -- Fix jobs getting rejected when submitting with --gpus option from older > versions of job submission commands (23.02 and older). > -- cgroup/v2 - Return 0 for VSZ. Kernel cgroups do not provide this metric. > -- scrun - Avoid race condition where outbound RPCs could be corrupted. > -- scrun - Avoid race condition that could cause a crash while compiled in > debug mode. > -- gpu/rsmi - Disable gpu usage statistics when not using ROCM 6.0.0+ > -- Fix stuck processes and incorrect environment when using --get-user-env. > -- Avoid segfault in the slurmdbd when TrackWCKey=no but you are still using > use WCKeys. > -- Fix ctld segfault with TopologyParam=RoutePart and no partition defined. > -- slurmctld - Fix missing --deadline handling for jobs not evaluated by the > schedulers (i.e. non-runnable, skipped for other reasons, etc.). > -- Demote some eio related logs from error to verbose in user commands. These > are not generally actionable by the user and are easilly generated by port > scanning a machine running srun. > -- Make sprio correctly print array tasks that have not yet been split out. > -- topology/block - Restrict the number of last-level blocks in any allocation. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

RHEL 8.9+SLURM-23.11.3+MLNX_OFED_LINUX-23.10-1.1.9.0+ OpenMPI-5.0.2
by S L 20 Feb '24

20 Feb '24

Hi All, We're currently in the process of setting up SLURM on a RHEL 8.9 based cluster. Here's a summary of the steps we've taken so far: Installed MLNX OFED ConnectX-5.2. Compiled and installed PMiX and UCX. Compiled and installed Slurm with PMiX_v4 and UCX support. Compiled OpenMPI with SLURM, PMIx, libevent, and hwloc support. All compute nodes are reachable via the IB network. *Problem:* While hello world MPI jobs are working fine on multiple nodes, the jobs are not utilizing Infiniband. srun --mpi=pmix -N2 -n2 --ntasks-per-node=2 ./hello > log.out 2>&1 Output from srun --mpi=list: MPI plugin types are... none cray_shasta pmi2 pmix specific pmix plugin versions available: pmix_v4 Could someone please point me in the right direction on how to troubleshoot this issue? Thank you for your assistance. Sudhakar

2 1

URL for how to do for SLURM accounting setup
by John Joseph 20 Feb '24

20 Feb '24

Dear ALL, Good morning we were able to setup a test SLURM based system, with 4 nodes , Ubuntu 22.04 LTS and we were able to run COMSOL using "comsol batch" command Now we plan to have accounting https://slurm.schedmd.com/accounting.html Like to reach out and get guidance on any tutorial or how to do documentation on setting up accounting Appreciate your support Thanks Joseph John

2 2

CPU utilisation using two commands scontrol association and sreport makes a huge difference
by prachikakade.lit＠gmail.com 20 Feb '24

20 Feb '24

Dear Team, I created a small cluster of 3 nodes on my VM ware to work on the CPU utilization concept. I created a user name= hpcuser01, and allocated GrpTresMin=cpu=59000040 -> CPU minutes and gpu=0 Now, when I checked his utilization using scontrol association cmd # scontrol show ass user=hpcuser01 | grep -n15 hpcuser01 "" Output: QOS=hpcuser01_test(11) UsageRaw=5789263754.153907 GrpJobs=N(0) GrpJobsAccrue=N(5) GrpSubmitJobs=N(5) GrpWall=N(1188290.27) GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),billing=N(0),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0) GrpTRESMins=cpu=59000040(59001152),mem=N(293429427523),energy=N(0),node=N(1993778),billing=N(96073808),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=1(0) GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=46080(0),billing=N(0),fs/disk=N(0),vmem=N(0),pages=N(0)"" Here we can see that GrpTRESMins=cpu=59000040(59001152) after converting in Hours ( value /60) it will be: 983334(983352) CPU hours This means 983352 CPU hours have been used by hpcuser01 but, when I checked his utilization using sreport cmd #sreport -t hour --tres=cpu,gres/gpu,billing cluster AccountUtilizationByUser user=hpcuser01 start=2021-01-01T00:00:00 end=2024-02-20T23:59:59 -P | grep -iw -e billing -e cpu -e gres/gpu Output: cpu|979403 billing|1594902 gres/gpu|0 here I found that his utilisation is : 979403 I could like to understand why there is so much of a difference using two cmds to find out utilisation. Almost the difference is 979403 - 983352 = 3949 Request you to help me out with this. Regards, Prachi

1 0

Slurm Power Saving Guide: Why doesnt slurm mark as failed when resumeProgram returns =/= 0
by Xaver Stiensmeier 19 Feb '24

19 Feb '24

Dear slurm-user list, I had cases where our resumeProgram failed due to temporary cloud timeouts. In that case the resumeProgram returns a value =/= 0. Why does Slurm still wait until resumeTimeout instead of just accepting the startup as failed which then should lead to a rescheduling of the job. Is there some way to achieve the described effect i.e. tell Slurm: "You can stop waiting, the node won't come alive." or am I missing the correct way how this should be handled in Slurm? Best regards, Xaver

1 0

slurmdbd 17.02: "cluster not registered" (but things work)
by Matthias Leopold 19 Feb '24

19 Feb '24

Hi, I need to take care of a 17.02 Slurm cluster (I'm preparing it for upgrades). I see that slurmdbd logs various "cluster not registered" messages at startup (DBD_CLUSTER_TRES,DBD_JOB_START,DBD_STEP_START), but I don't see a real problem. Accounting works. Do I have to worry? Can this be related to upper/lower case issues with ClusterName? For some reason ClusterName is all upper case in slurm.conf. According to docs (and my experience with Slurm 21.08) though, this should be OK. Thx for help Matthias

1 0