March 2024 - slurm-users - lists.schedmd.com

Interfaces of topology/tree and Topology Awareness
by nico.derl＠tutanota.com 15 Apr '24

15 Apr '24

Hello everyone, I'm trying to improve topology awareness in a local Slurm-managed HPC system. It's using the default hierarchical 3-level topology with the tree-plugin. It however does not always confine jobs to the most tightly packed group of nodes, seems to over-provision switches for smaller jobs, and gets slow or overwhelmed with jobs that have a high node count. I'd like to implement something more literally aligned with best-fit, but I'm having trouble understanding the relevant interfaces to hook into the topology model of Slurm. I would like a high-level explanation of how the tree- and common topology components work, how they integrate into the higher scheduling logic and what the internal topology model looks like. Or some pointers to relevant docs discussing this. I have read the topology guide and its dev-doc, which does note some of the caveats I mentioned. It however only talks about providing a set of weights to the upper logic levels in the form of a node ranking. I can't see how this ranking resembles the topology and how it's being used. From looking at the signatures and C-code I can tell this much: topology-tree consumes the topology.conf and generates a ranking of some kind that is passed to topology-common. topology-common consumes a ranking and uses its own gres-sched to figure out what nodes can fit a job (possibly pulling info from the gres-select-plugin to determine node capabilities). It's then supposed to apply a best-fit algorithm to efficiently fill up vacant cluster-capacity, but I can't manage to follow this part in the code as everything crumbles into separate files that I can't link correctly in my head. Thanks in advance. referenced docs: <https://slurm.schedmd.com/topology.html> <https://hpc.rz.rptu.de/documentation/topology_plugin.html> <https://github.com/SchedMD/slurm/tree/master/src/plugins/topology/common>

2 1

PMIxDirectConn Setting
by efweber＠uwe.nsa.gov 28 Mar '24

28 Mar '24

Hi, The documentation for mpi.conf says: PMIxDirectConn={true|false} Disable direct launching of tasks. Default is "true". I'm finding this a bit confusing. Does true really mean that Direct Connection is disabled? I would have thought true meant enabled. Thanks

1 0

Incorrect hyperthreading with Slurm 23.11
by Guillaume COCHARD 28 Mar '24

28 Mar '24

Hello, We have upgraded our cluster to Slurm 23.11.1 then, a few weeks later, to 23.11.4. Since then, Slurm doesn't detect hyperthreaded CPUs. We have downgraded our test cluster, the issue is not present with Slurm 22.05 (we had skipped Slurm 23.02). For example, we are working with this node: $ slurmd -C NodeName=node03 CPUs=40 Boards=1 SocketsPerBoard=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=128215 It is defined like this in slurm.conf: SelectTypeParameters=CR_CPU_Memory TaskPlugin=task/cgroup,task/affinity NodeName=node03 CPUs=40 RealMemory=150000 Feature=htc MemSpecLimit=5000 NodeSet=htc Feature=htc PartitionName=htc Default=YES MinNodes=0 MaxNodes=1 Nodes=htc DefMemPerCPU=1000 State=UP LLN=Yes MaxMemPerNode=142000 So no oversubscribing, 20 cores and 40 CPUs thanks to hyperthreading. Until the updgrade, Slurm was allocating those 40 CPUs: when launching 40 jobs of 1 CPU, each of those job would use one 1 CPU. This is the expected behavior. Since the upgrade, we can still launch those 40 jobs, but only the first half of the CPUs will be used (CPUs 0 to 19 according to htop). Each of those CPUs is used by 2 jobs, and the second half of the CPUs (#20 to 39) stay completely idle. When launching 40 stress processes directly in the node without using Slurm all the CPUs are used. When allocating a specific CPU with srun, it works until CPU #19 and then an error occurs even if the allocation includes all the CPUs of the node: #SBATCH --ntasks=1 #SBATCH --cpus-per-task=40 # Works for 0 to 19 srun --cpu-bind=v,map_cpu:19 stress.py # Doesn't work (20 to 39) srun --cpu-bind=v,map_cpu:20 stress.py # Output: srun: error: CPU binding outside of job step allocation, allocated CPUs are: 0x00000FFFFF. srun: error: Task launch for StepId=57194.0 failed on node node03: Unable to satisfy cpu bind request srun: error: Application launch failed: Unable to satisfy cpu bind request srun: Job step aborted This behaviour concerns all our nodes, some of which have been restarted recently and others have not. This causes the jobs to be frequently interrupted, augmenting the difference between the system real time and user+system times and making the jobs slower. We have been peering the documentation but, from what we understand, our configuration seems correct. In particular, as advised by the documentation[1], we don't set up ThreadsPerCore in slurm.conf. Are we missing something, or is there a regression or a change in Slurm configuration since the version 23.11? Thank you, Guillaume [1] : https://slurm.schedmd.com/slurm.conf.html#OPT_ThreadsPerCore

2 1

FairShare priority questions
by Long, Daniel S. 27 Mar '24

27 Mar '24

Hi, I’m trying to set up multifactor priority on our cluster and am having some trouble getting it to behave the way I’d like. My main issues seem to revolve around FairShare. We have multiple projects on our cluster and multiple users in those projects (and some users are in multiple projects, of course). I would like the FairShare to be based only on the project associated with the job; if user A and user B both submit jobs on project C, the FairShare should be identical. However, it looks like the FairShare is based on both the project and the user. Is there a way to get the behavior I’m looking for? Thanks for any help you can provide.

3 2

Slurm releases move to a six-month cycle
by Tim Wickberg 26 Mar '24

26 Mar '24

Slurm major releases are moving to a six month release cycle. This change starts with the upcoming Slurm 24.05 release this May. Slurm 24.11 will follow in November 2024. Major releases then continue every May and November in 2025 and beyond. There are two main goals of this change: - Faster delivery of newer features and functionality for customers. - "Predictable" release timing, especially for those sites that would prefer to upgrade during an annual system maintenance window. SchedMD will be adjusting our handling of backwards-compatibility within Slurm itself, and how SchedMD's support services will handle older releases. For the 24.05 release, Slurm will still only support upgrading from (and mixed-version operations with) the prior two releases (23.11, 23.02). Starting with 24.11, Slurm will start supporting upgrades from the prior three releases (24.05, 23.11, 23.02). SchedMD's Slurm Support has been built around an 18-month cycle. This 18-month cycle has traditionally covered the current stable release, plus one prior major releases. With the increase in release frequency this support window will now cover to the current stable release, plus two prior major releases. The blog post version of this announcement includes a table that outlines the updated support lifecycle: https://www.schedmd.com/slurm-releases-move-to-a-six-month-cycle/ - Tim -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

controller backup slurmctld error while takeover
by Miriam Olmi 26 Mar '24

26 Mar '24

Dear all, I am having trouble finalizing the configuration of the backup controller for my slurm cluster. In principle, if no job is running everything seems fine: both the slurmctld services on the primary and the backup controller are running and if I stop the service on the primary controller after 10s more or less (SlurmctldTimeout = 10 sec) the backup controller takes over. Also, if I run the sinfo or squeue command during the 10s of inactivity, the shell stay pending but it recover perfectly after the time needed by the backup controller to take control and it works the same when the primary controller is back. Unfortunately, if I try to do the same test while a job is running there are two different behaviors depending on the initial scenario. 1st scenario: Both the primary and backup controller are fine. I launch a batch script and I verify the script is running with sinfo and squeue. While the script is still running I stop the service on the primary controller with success but at this point everything gets crazy: on the backup controller in the slurmctld service log I find the following errors: slurmctld: error: Invalid RPC received REQUEST_JOB_INFO while in standby mode slurmctld: error: Invalid RPC received REQUEST_PARTITION_INFO while in standby mode slurmctld: error: Invalid RPC received REQUEST_JOB_INFO while in standby mode slurmctld: error: Invalid RPC received REQUEST_PARTITION_INFO while in standby mode slurmctld: error: slurm_accept_msg_conn poll: Bad address slurmctld: error: slurm_accept_msg_conn poll: Bad address and the commands sinfo and squeue are Unable to contact slurm controller (connect failure). 2nd scenario: the primary controller is stopped and I launch a batch job while the backup controller is the only one working. While the job is running, I restart the slurmctld service on the primary controller. In this case the primary controller takes over immediately: everything is smooth and safe and the sinfo and squeue commands continue to work perfectly. What might be the problem? Many thanks in advance! Miriam

2 5

cgroups_exporter for slurm on rhel9 (cgroups-v2)
by Saluja, Prabhjyot 25 Mar '24

25 Mar '24

Hi All, We are currently trying to set up cgroup_exporter <https://github.com/treydock/cgroup_exporter> for slurm. It's been working smoothly with cgroups.v1 and slurm-22.05.7. However, we're facing some challenges with RHEL-9, slurm-23.11.1 and cgroups.v2. The cgroup_exporter isn't capturing the slurm cgroup job information. I'm reaching out to see if any other sites have managed to make this work. If you're using a different exporter that's working for your site, could you please let us know? Thanks! Regards, Singh

2 1

Slurm version 23.11.5 is now available
by Tim McMullan 21 Mar '24

21 Mar '24

We are pleased to announce the availability of Slurm version 23.11.5. The 23.11.5 release includes some important fixes related to newer features as well as some database fixes. The most noteworthy fixes include fixing the sattach command (which only worked for root and SlurmUser after 23.11.0) and fixing an issue while constructing the new lineage database entries. This last change will also perform a query during the upgrade from any prior 23.11 version to fix existing databases. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.5 > ========================== > -- Fix Debian package build on systems that are not able to query the systemd > package. > -- data_parser/v0.0.40 - Emit a warning instead of an error if a disabled > parser is invoked. > -- slurmrestd - Improve handling when content plugins rely on parsers > that haven't been loaded. > -- Fix old pending jobs dying (Slurm version 21.08.x and older) when upgrading > Slurm due to "Invalid message version" errors. > -- Have client commands sleep for progressively longer periods when backed off > by the RPC rate limiting system. > -- slurmctld - Ensure agent queue is flushed correctly at shutdown time. > -- slurmdbd - correct lineage construction during assoc table conversion for > partition based associations. > -- Add new RPCs and API call for faster querying of job states from slurmctld. > -- slurmrestd - Add endpoint '/slurm/{data_parser}/jobs/state'. > -- squeue - Add `--only-job-state` argument to use faster query of job states. > -- Make a job requesting --no-requeue, or JobRequeue=0 in the slurm.conf, > supersede RequeueExit[Hold]. > -- Add sackd man page to the Debian package. > -- Fix issues with tasks when a job was shrinked more than once. > -- Fix reservation update validation that resulted in reject of correct > updates of reservation when the reservation was running jobs. > -- Fix possible segfault when the backup slurmctld is asserting control. > -- Fix regression introduced in 23.02.4 where slurmctld was not properly > tracking the total GRES selected for exclusive multi-node jobs, potentially > and incorrectly bypassing limits. > -- Fix tracking of jobs typeless GRES count when multiple typed GRES with the > same name are also present in the job allocation. Otherwise, the job could > bypass limits configured for the typeless GRES. > -- Fix tracking of jobs typeless GRES count when request specification has a > typeless GRES name first and then typed GRES of different names (i.e. > --gres=gpu:1,tmpfs:foo:2,tmpfs:bar:7). Otherwise, the job could bypass > limits configured for the generic of the typed one (tmpfs in the example). > -- Fix batch step not having SLURM_CLUSTER_NAME filled in. > -- slurmstepd - Avoid error during `--container` job cleanup about > RunTimeQuery never being configured. Results in cleanup where job steps not > fully started. > -- Fix nodes not being rebooted when using salloc/sbatch/srun "--reboot" flag. > -- Send scrun.lua in configless mode. > -- Fix rejecting an interactive job whose extra constraint request cannot > immediately be satisfied. > -- Fix regression in 23.11.0 when parsing LogTimeFormat=iso8601_ms that > prevented milliseconds from being printed. > -- Fix issue where you could have a gpu allocated as well as a shard on that > gpu allocated at the same time. > -- Fix slurmctld crashes when using extra constraints with job arrays. > -- sackd/slurmrestd/scrun - Avoid memory leak on new unix socket connection. > -- The failed node field is filled when a node fails but does not time out. > -- slurmrestd - Remove requiring job script field and job component script > fields to both be populated in the `POST /slurm/v0.0.40/job/submit` > endpoint as there can only be one batch step script for a job. > -- slurmrestd - When job script is provided in '.jobs[].script' and '.script' > fields, the '.script' field's value will be used in the > `POST /slurm/v0.0.40/job/submit` endpoint. > -- slurmrestd - Reject HetJob submission missing or empty batch script for > first Het component in the `POST /slurm/v0.0.40/job/submit` endpoint. > -- slurmrestd - Reject job when empty batch script submitted to the > POST /slurm/v0.0.40/job/submit` endpoint. > -- Fix pam_slurm and pam_slurm_adopt when using auth/slurm. > -- slurmrestd - Add 'cores_per_socket' field to > `POST /slurm/v0.0.40/job/submit` endpoint. > -- Fix srun and other Slurm commands running within a "configless" salloc when > salloc itself fetched the config. > -- Enforce binding with shared gres selection if requested. > -- Fix job allocation failures when the requested tres type or name ends in > "gres" or "license". > -- accounting_storage/mysql - Fix lineage string construction when adding a > user association with a partition. > -- Fix sattach command. > -- Fix ReconfigFlags. Due how reconfig was changed in 23.11, they will also > be used to influence the slurmctld startup as well. > -- Fix starting slurmd in configless mode if MUNGE support was disabled. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Re: Lua script
by Gestió Servidors 21 Mar '24

21 Mar '24

Hello, I answer about my question: * What is the contents of your /etc/slurm/job_submit.lua file? function slurm_job_submit(job_desc, part_list, submit_uid) if (job_desc.user_id == 1008) then slurm.log_info("Trabajo sometido por druiz") if (job_desc['partition'] == "nodo.q") then if (job_desc['time_limit'] > 345600) # 345600 seconds == 4 days # nodo.q partition has "PartitionName=nodo.q Nodes=clus[01-12] Default=YES MaxTime=04:00:00" configuration return slurm.FAILURE end end end return slurm.SUCCESS end slurm.log_info("initialized") return slurm.SUCCESS * Did you reconfigure slurmctld? Yes, in the first time, I ran "scontrol reconfigure", but after checking that limits wasn't applyed, I restarted slurmctld daemon * Check the log file by: grep job_submit /var/log/slurm/slurmctld.log In my slurmctld.log file in SLURM server, "grep job_submit /var/log/slurm/slurmctld.log" doesn't return anything... * What is your Slurm version? 23.11.0 Thanks.

1 0

Re: Lua script
by Gestió Servidors 20 Mar '24

20 Mar '24

Hello, after adding "EnforcePartLimits=ALL" in slurm.conf and restarting slurmctld daemon, job continues being accepted... so I don't undertand where I'm doing some wrong. My slurm.conf is this: ControlMachine=my_server MailProg=/bin/mail MpiDefault=none ProctrackType=proctrack/linuxproc ReturnToService=2 SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm SlurmdUser=root AuthType=auth/munge StateSaveLocation=/var/log/slurm SwitchType=switch/none TaskPlugin=task/none,task/affinity,task/cgroup TaskPluginParam=none DebugFlags=NO_CONF_HASH,Backfill,BackfillMap,SelectType,Steps,TraceJobs JobSubmitPlugins=lua SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core SchedulerParameters=max_script_size=20971520 EnforcePartLimits=ALL CoreSpecPlugin=core_spec/none AccountingStorageType=accounting_storage/slurmdbd AccountingStoreFlags=job_comment JobCompType=jobcomp/filetxt JobCompLoc=/var/log/slurm/job_completions ClusterName=my_cluster JobAcctGatherType=jobacct_gather/linux SlurmctldDebug=5 SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=5 SlurmdLogFile=/var/log/slurmd.log AccountingStorageEnforce=limits AccountingStorageHost=my_server NodeName=clus[01-06] CPUs=12 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 RealMemory=128387 TmpDisk=81880 Feature=big-mem NodeName=clus[07-12] CPUs=12 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 RealMemory=15491 TmpDisk=81880 Feature=small-mem NodeName=clus-login CPUs=4 SocketsPerBoard=2 CoresperSocket=2 ThreadsperCore=1 RealMemory=15886 TmpDisk=30705 PartitionName=nodo.q Nodes=clus[01-12] Default=YES MaxTime=04:00:00 State=UP AllocNodes=clus-login,clus05 MaxCPUsPerNode=12 KillOnBadExit=1 OverTimeLimit=30 # si el trabajo dura mas de 30 minutos despues del tiempo maximo (2 horas), se cancela TCPTimeout=5 PriorityType=priority/multifactor PriorityDecayHalfLife=7-0 PriorityCalcPeriod=5 PriorityUsageResetPeriod=QUARTERLY PriorityFavorSmall=NO PriorityMaxAge=7-0 PriorityWeightAge=10000 PriorityWeightFairshare=1000000 PriorityWeightJobSize=1000 PriorityWeightPartition=1000 PriorityWeightQOS=0 PropagateResourceLimitsExcept=MEMLOCK And testing script is this: #!/bin/bash #SBATCH --time=5-00:00:00 srun /bin/hostname date sleep 50 date Why my job is being submited into the queue and not refused BEFORE being queued? Thanks.

2 1

2024

slurm-users March 2024 ----- 2024 ----- May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users March 2024