March 2024 - slurm-users - lists.schedmd.com

job_container/tmpfs and srun.
by Phill Harvey-Smith 17 Jan '25

17 Jan '25

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

2 1

AllowAccounts partition setting
by Marko Markoc 16 Oct '24

16 Oct '24

Hi All, At one point, when we were running slurm 22, we set up a few new partitions with `AllowAccounts` setting enabled. We limit access to these partitions only to a few accounts. We manually create these accounts and users using the `sacctmgr` command. At that time, everything was working. Only users in these accounts were able to run jobs in that partition. Few months ago we upgraded slurm to 23.02.7 but just a couple of days ago we noticed that anyone could submit jobs to these partitions. In addition to that, you can provide any value to the `--account` directive and slurm will accept and record it in the job accounting database. In our initial testing, you could only provide `--account` value that has been previously created. I'm still looking into this but I wanted to check if there was any change in version 23 that could have impacted this? I don't remember seeing anything in release notes. Thanks, Marko

6 8

Can't schedule on cloud node: State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING
by Xaver Stiensmeier 20 Sep '24

20 Sep '24

Dear slurm-user list, I have a cloud node that is powered up and down on demand. Rarely it can happen that slurm's resumeTimeout is reached and the node is therefore powered down. We have set ReturnToService=2 in order to avoid the node being marked down, because the instance behind that node is created on demand and therefore after a failure nothing stops the system to start the node again as it is a different instance. I thought this would be enough, but apparently the node is still marked with "NOT_RESPONDING" which leads to slurm not trying to schedule on it. After a while NOT_RESPONDING is removed, but I would like to move it directly from within my fail script if possible so that the node can return to service immediately and not be blocked by "NOT_RESPONDING". Best regards, Xaver

2 3

Interfaces of topology/tree and Topology Awareness
by nico.derl＠tutanota.com 15 Apr '24

15 Apr '24

Hello everyone, I'm trying to improve topology awareness in a local Slurm-managed HPC system. It's using the default hierarchical 3-level topology with the tree-plugin. It however does not always confine jobs to the most tightly packed group of nodes, seems to over-provision switches for smaller jobs, and gets slow or overwhelmed with jobs that have a high node count. I'd like to implement something more literally aligned with best-fit, but I'm having trouble understanding the relevant interfaces to hook into the topology model of Slurm. I would like a high-level explanation of how the tree- and common topology components work, how they integrate into the higher scheduling logic and what the internal topology model looks like. Or some pointers to relevant docs discussing this. I have read the topology guide and its dev-doc, which does note some of the caveats I mentioned. It however only talks about providing a set of weights to the upper logic levels in the form of a node ranking. I can't see how this ranking resembles the topology and how it's being used. From looking at the signatures and C-code I can tell this much: topology-tree consumes the topology.conf and generates a ranking of some kind that is passed to topology-common. topology-common consumes a ranking and uses its own gres-sched to figure out what nodes can fit a job (possibly pulling info from the gres-select-plugin to determine node capabilities). It's then supposed to apply a best-fit algorithm to efficiently fill up vacant cluster-capacity, but I can't manage to follow this part in the code as everything crumbles into separate files that I can't link correctly in my head. Thanks in advance. referenced docs: <https://slurm.schedmd.com/topology.html> <https://hpc.rz.rptu.de/documentation/topology_plugin.html> <https://github.com/SchedMD/slurm/tree/master/src/plugins/topology/common>

2 1

PMIxDirectConn Setting
by efweber＠uwe.nsa.gov 28 Mar '24

28 Mar '24

Hi, The documentation for mpi.conf says: PMIxDirectConn={true|false} Disable direct launching of tasks. Default is "true". I'm finding this a bit confusing. Does true really mean that Direct Connection is disabled? I would have thought true meant enabled. Thanks

1 0

Incorrect hyperthreading with Slurm 23.11
by Guillaume COCHARD 27 Mar '24

27 Mar '24

Hello, We have upgraded our cluster to Slurm 23.11.1 then, a few weeks later, to 23.11.4. Since then, Slurm doesn't detect hyperthreaded CPUs. We have downgraded our test cluster, the issue is not present with Slurm 22.05 (we had skipped Slurm 23.02). For example, we are working with this node: $ slurmd -C NodeName=node03 CPUs=40 Boards=1 SocketsPerBoard=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=128215 It is defined like this in slurm.conf: SelectTypeParameters=CR_CPU_Memory TaskPlugin=task/cgroup,task/affinity NodeName=node03 CPUs=40 RealMemory=150000 Feature=htc MemSpecLimit=5000 NodeSet=htc Feature=htc PartitionName=htc Default=YES MinNodes=0 MaxNodes=1 Nodes=htc DefMemPerCPU=1000 State=UP LLN=Yes MaxMemPerNode=142000 So no oversubscribing, 20 cores and 40 CPUs thanks to hyperthreading. Until the updgrade, Slurm was allocating those 40 CPUs: when launching 40 jobs of 1 CPU, each of those job would use one 1 CPU. This is the expected behavior. Since the upgrade, we can still launch those 40 jobs, but only the first half of the CPUs will be used (CPUs 0 to 19 according to htop). Each of those CPUs is used by 2 jobs, and the second half of the CPUs (#20 to 39) stay completely idle. When launching 40 stress processes directly in the node without using Slurm all the CPUs are used. When allocating a specific CPU with srun, it works until CPU #19 and then an error occurs even if the allocation includes all the CPUs of the node: #SBATCH --ntasks=1 #SBATCH --cpus-per-task=40 # Works for 0 to 19 srun --cpu-bind=v,map_cpu:19 stress.py # Doesn't work (20 to 39) srun --cpu-bind=v,map_cpu:20 stress.py # Output: srun: error: CPU binding outside of job step allocation, allocated CPUs are: 0x00000FFFFF. srun: error: Task launch for StepId=57194.0 failed on node node03: Unable to satisfy cpu bind request srun: error: Application launch failed: Unable to satisfy cpu bind request srun: Job step aborted This behaviour concerns all our nodes, some of which have been restarted recently and others have not. This causes the jobs to be frequently interrupted, augmenting the difference between the system real time and user+system times and making the jobs slower. We have been peering the documentation but, from what we understand, our configuration seems correct. In particular, as advised by the documentation[1], we don't set up ThreadsPerCore in slurm.conf. Are we missing something, or is there a regression or a change in Slurm configuration since the version 23.11? Thank you, Guillaume [1] : https://slurm.schedmd.com/slurm.conf.html#OPT_ThreadsPerCore

2 1

FairShare priority questions
by Long, Daniel S. 27 Mar '24

27 Mar '24

Hi, I’m trying to set up multifactor priority on our cluster and am having some trouble getting it to behave the way I’d like. My main issues seem to revolve around FairShare. We have multiple projects on our cluster and multiple users in those projects (and some users are in multiple projects, of course). I would like the FairShare to be based only on the project associated with the job; if user A and user B both submit jobs on project C, the FairShare should be identical. However, it looks like the FairShare is based on both the project and the user. Is there a way to get the behavior I’m looking for? Thanks for any help you can provide.

3 2

Slurm releases move to a six-month cycle
by Tim Wickberg 26 Mar '24

26 Mar '24

Slurm major releases are moving to a six month release cycle. This change starts with the upcoming Slurm 24.05 release this May. Slurm 24.11 will follow in November 2024. Major releases then continue every May and November in 2025 and beyond. There are two main goals of this change: - Faster delivery of newer features and functionality for customers. - "Predictable" release timing, especially for those sites that would prefer to upgrade during an annual system maintenance window. SchedMD will be adjusting our handling of backwards-compatibility within Slurm itself, and how SchedMD's support services will handle older releases. For the 24.05 release, Slurm will still only support upgrading from (and mixed-version operations with) the prior two releases (23.11, 23.02). Starting with 24.11, Slurm will start supporting upgrades from the prior three releases (24.05, 23.11, 23.02). SchedMD's Slurm Support has been built around an 18-month cycle. This 18-month cycle has traditionally covered the current stable release, plus one prior major releases. With the increase in release frequency this support window will now cover to the current stable release, plus two prior major releases. The blog post version of this announcement includes a table that outlines the updated support lifecycle: https://www.schedmd.com/slurm-releases-move-to-a-six-month-cycle/ - Tim -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

controller backup slurmctld error while takeover
by Miriam Olmi 26 Mar '24

26 Mar '24

Dear all, I am having trouble finalizing the configuration of the backup controller for my slurm cluster. In principle, if no job is running everything seems fine: both the slurmctld services on the primary and the backup controller are running and if I stop the service on the primary controller after 10s more or less (SlurmctldTimeout = 10 sec) the backup controller takes over. Also, if I run the sinfo or squeue command during the 10s of inactivity, the shell stay pending but it recover perfectly after the time needed by the backup controller to take control and it works the same when the primary controller is back. Unfortunately, if I try to do the same test while a job is running there are two different behaviors depending on the initial scenario. 1st scenario: Both the primary and backup controller are fine. I launch a batch script and I verify the script is running with sinfo and squeue. While the script is still running I stop the service on the primary controller with success but at this point everything gets crazy: on the backup controller in the slurmctld service log I find the following errors: slurmctld: error: Invalid RPC received REQUEST_JOB_INFO while in standby mode slurmctld: error: Invalid RPC received REQUEST_PARTITION_INFO while in standby mode slurmctld: error: Invalid RPC received REQUEST_JOB_INFO while in standby mode slurmctld: error: Invalid RPC received REQUEST_PARTITION_INFO while in standby mode slurmctld: error: slurm_accept_msg_conn poll: Bad address slurmctld: error: slurm_accept_msg_conn poll: Bad address and the commands sinfo and squeue are Unable to contact slurm controller (connect failure). 2nd scenario: the primary controller is stopped and I launch a batch job while the backup controller is the only one working. While the job is running, I restart the slurmctld service on the primary controller. In this case the primary controller takes over immediately: everything is smooth and safe and the sinfo and squeue commands continue to work perfectly. What might be the problem? Many thanks in advance! Miriam

2 5

cgroups_exporter for slurm on rhel9 (cgroups-v2)
by Saluja, Prabhjyot 25 Mar '24

25 Mar '24

Hi All, We are currently trying to set up cgroup_exporter <https://github.com/treydock/cgroup_exporter> for slurm. It's been working smoothly with cgroups.v1 and slurm-22.05.7. However, we're facing some challenges with RHEL-9, slurm-23.11.1 and cgroups.v2. The cgroup_exporter isn't capturing the slurm cgroup job information. I'm reaching out to see if any other sites have managed to make this work. If you're using a different exporter that's working for your site, could you please let us know? Thanks! Regards, Singh

2 1

Slurm version 23.11.5 is now available
by Tim McMullan 21 Mar '24

21 Mar '24

We are pleased to announce the availability of Slurm version 23.11.5. The 23.11.5 release includes some important fixes related to newer features as well as some database fixes. The most noteworthy fixes include fixing the sattach command (which only worked for root and SlurmUser after 23.11.0) and fixing an issue while constructing the new lineage database entries. This last change will also perform a query during the upgrade from any prior 23.11 version to fix existing databases. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.5 > ========================== > -- Fix Debian package build on systems that are not able to query the systemd > package. > -- data_parser/v0.0.40 - Emit a warning instead of an error if a disabled > parser is invoked. > -- slurmrestd - Improve handling when content plugins rely on parsers > that haven't been loaded. > -- Fix old pending jobs dying (Slurm version 21.08.x and older) when upgrading > Slurm due to "Invalid message version" errors. > -- Have client commands sleep for progressively longer periods when backed off > by the RPC rate limiting system. > -- slurmctld - Ensure agent queue is flushed correctly at shutdown time. > -- slurmdbd - correct lineage construction during assoc table conversion for > partition based associations. > -- Add new RPCs and API call for faster querying of job states from slurmctld. > -- slurmrestd - Add endpoint '/slurm/{data_parser}/jobs/state'. > -- squeue - Add `--only-job-state` argument to use faster query of job states. > -- Make a job requesting --no-requeue, or JobRequeue=0 in the slurm.conf, > supersede RequeueExit[Hold]. > -- Add sackd man page to the Debian package. > -- Fix issues with tasks when a job was shrinked more than once. > -- Fix reservation update validation that resulted in reject of correct > updates of reservation when the reservation was running jobs. > -- Fix possible segfault when the backup slurmctld is asserting control. > -- Fix regression introduced in 23.02.4 where slurmctld was not properly > tracking the total GRES selected for exclusive multi-node jobs, potentially > and incorrectly bypassing limits. > -- Fix tracking of jobs typeless GRES count when multiple typed GRES with the > same name are also present in the job allocation. Otherwise, the job could > bypass limits configured for the typeless GRES. > -- Fix tracking of jobs typeless GRES count when request specification has a > typeless GRES name first and then typed GRES of different names (i.e. > --gres=gpu:1,tmpfs:foo:2,tmpfs:bar:7). Otherwise, the job could bypass > limits configured for the generic of the typed one (tmpfs in the example). > -- Fix batch step not having SLURM_CLUSTER_NAME filled in. > -- slurmstepd - Avoid error during `--container` job cleanup about > RunTimeQuery never being configured. Results in cleanup where job steps not > fully started. > -- Fix nodes not being rebooted when using salloc/sbatch/srun "--reboot" flag. > -- Send scrun.lua in configless mode. > -- Fix rejecting an interactive job whose extra constraint request cannot > immediately be satisfied. > -- Fix regression in 23.11.0 when parsing LogTimeFormat=iso8601_ms that > prevented milliseconds from being printed. > -- Fix issue where you could have a gpu allocated as well as a shard on that > gpu allocated at the same time. > -- Fix slurmctld crashes when using extra constraints with job arrays. > -- sackd/slurmrestd/scrun - Avoid memory leak on new unix socket connection. > -- The failed node field is filled when a node fails but does not time out. > -- slurmrestd - Remove requiring job script field and job component script > fields to both be populated in the `POST /slurm/v0.0.40/job/submit` > endpoint as there can only be one batch step script for a job. > -- slurmrestd - When job script is provided in '.jobs[].script' and '.script' > fields, the '.script' field's value will be used in the > `POST /slurm/v0.0.40/job/submit` endpoint. > -- slurmrestd - Reject HetJob submission missing or empty batch script for > first Het component in the `POST /slurm/v0.0.40/job/submit` endpoint. > -- slurmrestd - Reject job when empty batch script submitted to the > POST /slurm/v0.0.40/job/submit` endpoint. > -- Fix pam_slurm and pam_slurm_adopt when using auth/slurm. > -- slurmrestd - Add 'cores_per_socket' field to > `POST /slurm/v0.0.40/job/submit` endpoint. > -- Fix srun and other Slurm commands running within a "configless" salloc when > salloc itself fetched the config. > -- Enforce binding with shared gres selection if requested. > -- Fix job allocation failures when the requested tres type or name ends in > "gres" or "license". > -- accounting_storage/mysql - Fix lineage string construction when adding a > user association with a partition. > -- Fix sattach command. > -- Fix ReconfigFlags. Due how reconfig was changed in 23.11, they will also > be used to influence the slurmctld startup as well. > -- Fix starting slurmd in configless mode if MUNGE support was disabled. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Re: Lua script
by Gestió Servidors 21 Mar '24

21 Mar '24

Hello, I answer about my question: * What is the contents of your /etc/slurm/job_submit.lua file? function slurm_job_submit(job_desc, part_list, submit_uid) if (job_desc.user_id == 1008) then slurm.log_info("Trabajo sometido por druiz") if (job_desc['partition'] == "nodo.q") then if (job_desc['time_limit'] > 345600) # 345600 seconds == 4 days # nodo.q partition has "PartitionName=nodo.q Nodes=clus[01-12] Default=YES MaxTime=04:00:00" configuration return slurm.FAILURE end end end return slurm.SUCCESS end slurm.log_info("initialized") return slurm.SUCCESS * Did you reconfigure slurmctld? Yes, in the first time, I ran "scontrol reconfigure", but after checking that limits wasn't applyed, I restarted slurmctld daemon * Check the log file by: grep job_submit /var/log/slurm/slurmctld.log In my slurmctld.log file in SLURM server, "grep job_submit /var/log/slurm/slurmctld.log" doesn't return anything... * What is your Slurm version? 23.11.0 Thanks.

1 0

Re: Lua script
by Gestió Servidors 20 Mar '24

20 Mar '24

Hello, after adding "EnforcePartLimits=ALL" in slurm.conf and restarting slurmctld daemon, job continues being accepted... so I don't undertand where I'm doing some wrong. My slurm.conf is this: ControlMachine=my_server MailProg=/bin/mail MpiDefault=none ProctrackType=proctrack/linuxproc ReturnToService=2 SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm SlurmdUser=root AuthType=auth/munge StateSaveLocation=/var/log/slurm SwitchType=switch/none TaskPlugin=task/none,task/affinity,task/cgroup TaskPluginParam=none DebugFlags=NO_CONF_HASH,Backfill,BackfillMap,SelectType,Steps,TraceJobs JobSubmitPlugins=lua SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core SchedulerParameters=max_script_size=20971520 EnforcePartLimits=ALL CoreSpecPlugin=core_spec/none AccountingStorageType=accounting_storage/slurmdbd AccountingStoreFlags=job_comment JobCompType=jobcomp/filetxt JobCompLoc=/var/log/slurm/job_completions ClusterName=my_cluster JobAcctGatherType=jobacct_gather/linux SlurmctldDebug=5 SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=5 SlurmdLogFile=/var/log/slurmd.log AccountingStorageEnforce=limits AccountingStorageHost=my_server NodeName=clus[01-06] CPUs=12 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 RealMemory=128387 TmpDisk=81880 Feature=big-mem NodeName=clus[07-12] CPUs=12 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 RealMemory=15491 TmpDisk=81880 Feature=small-mem NodeName=clus-login CPUs=4 SocketsPerBoard=2 CoresperSocket=2 ThreadsperCore=1 RealMemory=15886 TmpDisk=30705 PartitionName=nodo.q Nodes=clus[01-12] Default=YES MaxTime=04:00:00 State=UP AllocNodes=clus-login,clus05 MaxCPUsPerNode=12 KillOnBadExit=1 OverTimeLimit=30 # si el trabajo dura mas de 30 minutos despues del tiempo maximo (2 horas), se cancela TCPTimeout=5 PriorityType=priority/multifactor PriorityDecayHalfLife=7-0 PriorityCalcPeriod=5 PriorityUsageResetPeriod=QUARTERLY PriorityFavorSmall=NO PriorityMaxAge=7-0 PriorityWeightAge=10000 PriorityWeightFairshare=1000000 PriorityWeightJobSize=1000 PriorityWeightPartition=1000 PriorityWeightQOS=0 PropagateResourceLimitsExcept=MEMLOCK And testing script is this: #!/bin/bash #SBATCH --time=5-00:00:00 srun /bin/hostname date sleep 50 date Why my job is being submited into the queue and not refused BEFORE being queued? Thanks.

2 1

Node CPU Allocation and Effective CPU Allocation for Running Jobs
by Muhammad Akhdhor 18 Mar '24

18 Mar '24

Hi Everyone, We have a SLURM cluster of three different types of nodes. One partition consists of nodes that have a large number of CPUs, 256 CPUs on each node. I'm trying to find out the current CPU allocation on some of those nodes but part of the information I gathered seems to be incorrect. If I use "*scontrol show node <node-name>*", I get this for the CPU info: RealMemory=450000 AllocMem=262144 FreeMem=235397 Sockets=2 Boards=1 State=ALLOCATED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A CPUAlloc=256 CPUEfctv=256 CPUTot=256 CPULoad=126.65 CfgTRES=cpu=256,mem=450000M,billing=256 AllocTRES=cpu=256,mem=256G However, when I tried to identify those jobs to which the node's CPUs have been allocated, and get a tally of the allocated CPUs, I can only see 128 CPUs that are effectively allocated on that node, based on the output of *squeue --state=R -o "%C %N".* So I don't quite understand why the running jobs on the nodes account for just 128, and not 256, CPU allocation even though scontrol reports 100% CPU allocation on the node. Could this be due to some misconfiguration, or a bug in the SLURM version we're running? We're running Version=23.02.4. The interesting thing is that we have six nodes that have similar specs, and all of them show up as allocated in the output of *sinfo*, but the running jobs on each node account for just 128 CPU allocation, as if they're all capped at 128. Any thoughts, suggestions or assistance to figure this out would be greatly appreciated. Thanks, Muhammad

2 2

Slurm suspend preemption not working
by Verma, Nischey (HPC ENG,RAL,LSCI) 15 Mar '24

15 Mar '24

Hi All, we are trying to implement preemption in one of our partitions so we can run priority jobs on it and suspend the ones running on the partition and resume once the priority job is done. We have read through the Slurm documentation and did the configuration, but somehow we can not make it work. We tried other preemption like cancel which works fine, but Suspend isn't working. The jobs with higher priority which should suspend other jobs stays as pending and waiting for resources. We are using a QOS to assign priority to the job and also configuring the QOS so it can preempt certain other QOSs. this is the output of our scontrol show config : AccountingStorageBackupHost = (null) AccountingStorageEnforce = associations,limits,qos AccountingStorageHost = -Some-server- AccountingStorageExternalHost = (null) AccountingStorageParameters = (null) AccountingStoragePort = 6819 AccountingStorageTRES = cpu,mem,energy,node,billing,fs/disk,vmem,pages AccountingStorageType = accounting_storage/slurmdbd AccountingStorageUser = N/A AccountingStoreFlags = job_comment,job_env,job_extra,job_script AcctGatherEnergyType = (null) AcctGatherFilesystemType = (null) AcctGatherInterconnectType = (null) AcctGatherNodeFreq = 0 sec AcctGatherProfileType = (null) AllowSpecResourcesUsage = No AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge BatchStartTimeout = 10 sec BcastExclude = /lib,/usr/lib,/lib64,/usr/lib64 BcastParameters = (null) BOOT_TIME = 2024-03-15T13:32:05 BurstBufferType = (null) CliFilterPlugins = (null) ClusterName = cluster CommunicationParameters = (null) CompleteWait = 0 sec CoreSpecPlugin = (null) CpuFreqDef = Unknown CpuFreqGovernors = OnDemand,Performance,UserSpace CredType = cred/munge DebugFlags = (null) DefMemPerCPU = 3500 DependencyParameters = (null) DisableRootJobs = Yes EioTimeout = 60 EnforcePartLimits = ALL Epilog = (null) EpilogMsgTime = 2000 usec EpilogSlurmctld = (null) ExtSensorsType = (null) ExtSensorsFreq = 0 sec FairShareDampeningFactor = 1 FederationParameters = (null) FirstJobId = 1 GetEnvTimeout = 2 sec GresTypes = gpu GpuFreqDef = (null) GroupUpdateForce = 1 GroupUpdateTime = 600 sec HASH_VAL = Match HealthCheckInterval = 300 sec HealthCheckNodeState = ANY HealthCheckProgram = /usr/sbin/nhc InactiveLimit = 0 sec InteractiveStepOptions = --interactive --preserve-env --pty $SHELL JobAcctGatherFrequency = 30 JobAcctGatherType = jobacct_gather/linux JobAcctGatherParams = (null) JobCompHost = localhost JobCompLoc = (null) JobCompParams = (null) JobCompPort = 0 JobCompType = (null) JobCompUser = root JobContainerType = (null) JobDefaults = (null) JobFileAppend = 0 JobRequeue = 1 JobSubmitPlugins = lua KillOnBadExit = 0 KillWait = 30 sec LaunchParameters = (null) Licenses = (null) LogTimeFormat = iso8601_ms MailDomain = (null) MailProg = /bin/mail MaxArraySize = 1001 MaxBatchRequeue = 5 MaxDBDMsgs = 20024 MaxJobCount = 10000 MaxJobId = 67043328 MaxMemPerNode = UNLIMITED MaxNodeCount = 6 MaxStepCount = 40000 MaxTasksPerNode = 512 MCSPlugin = (null) MCSParameters = (null) MessageTimeout = 10 sec MinJobAge = 300 sec MpiDefault = pmix_v2 MpiParams = (null) NEXT_JOB_ID = 3626 NodeFeaturesPlugins = (null) OverTimeLimit = 0 min PluginDir = /usr/lib64/slurm PlugStackConfig = (null) PowerParameters = (null) PowerPlugin = (null) PreemptMode = GANG,SUSPEND PreemptParameters = (null) PreemptType = preempt/qos PreemptExemptTime = 00:00:00 PrEpParameters = (null) PrEpPlugins = prep/script PriorityParameters = (null) PrioritySiteFactorParameters = (null) PrioritySiteFactorPlugin = (null) PriorityDecayHalfLife = 7-00:00:00 PriorityCalcPeriod = 00:05:00 PriorityFavorSmall = No PriorityFlags = PriorityMaxAge = 7-00:00:00 PriorityUsageResetPeriod = NONE PriorityType = priority/multifactor PriorityWeightAge = 0 PriorityWeightAssoc = 0 PriorityWeightFairShare = 0 PriorityWeightJobSize = 0 PriorityWeightPartition = 0 PriorityWeightQOS = 5000 PriorityWeightTRES = (null) PrivateData = none ProctrackType = proctrack/cgroup Prolog = (null) PrologEpilogTimeout = 65534 PrologSlurmctld = (null) PrologFlags = (null) PropagatePrioProcess = 0 PropagateResourceLimits = ALL PropagateResourceLimitsExcept = (null) RebootProgram = /etc/slurm/slurmupdate.sh ReconfigFlags = (null) RequeueExit = (null) RequeueExitHold = (null) ResumeFailProgram = (null) ResumeProgram = (null) ResumeRate = 300 nodes/min ResumeTimeout = 60 sec ResvEpilog = (null) ResvOverRun = 0 min ResvProlog = (null) ReturnToService = 1 SchedulerParameters = bf_max_job_user=2 SchedulerTimeSlice = 30 sec SchedulerType = sched/backfill ScronParameters = (null) SelectType = select/cons_tres SelectTypeParameters = CR_CORE_MEMORY SlurmUser = slurm(888) SlurmctldAddr = (null) SlurmctldDebug = info SlurmctldHost[0] = -some-server- SlurmctldLogFile = /var/log/slurm/slurmctld.log SlurmctldPort = 6817 SlurmctldSyslogDebug = (null) SlurmctldPrimaryOffProg = (null) SlurmctldPrimaryOnProg = (null) SlurmctldTimeout = 120 sec SlurmctldParameters = (null) SlurmdDebug = info SlurmdLogFile = /var/log/slurm/slurmd.log SlurmdParameters = (null) SlurmdPidFile = /var/run/slurmd.pid SlurmdPort = 6818 SlurmdSpoolDir = /var/spool/slurm SlurmdSyslogDebug = (null) SlurmdTimeout = 300 sec SlurmdUser = root(0) SlurmSchedLogFile = (null) SlurmSchedLogLevel = 0 SlurmctldPidFile = /var/run/slurmctld.pid SLURM_CONF = /etc/slurm/slurm.conf SLURM_VERSION = 23.11.1 SrunEpilog = (null) SrunPortRange = 0-0 SrunProlog = (null) StateSaveLocation = /var/spool/slurmctld SuspendExcNodes = (null) SuspendExcParts = (null) SuspendExcStates = (null) SuspendProgram = (null) SuspendRate = 60 nodes/min SuspendTime = INFINITE SuspendTimeout = 30 sec SwitchParameters = (null) SwitchType = (null) TaskEpilog = (null) TaskPlugin = task/cgroup,task/affinity TaskPluginParam = none TaskProlog = (null) TCPTimeout = 2 sec TmpFS = /tmp TopologyParam = (null) TopologyPlugin = topology/default TrackWCKey = No TreeWidth = 16 UsePam = No UnkillableStepProgram = (null) UnkillableStepTimeout = 60 sec VSizeFactor = 0 percent WaitTime = 0 sec X11Parameters = (null) Cgroup Support Configuration: AllowedRAMSpace = 100.0% AllowedSwapSpace = 0.0% CgroupMountpoint = /sys/fs/cgroup CgroupPlugin = autodetect ConstrainCores = yes ConstrainDevices = yes ConstrainRAMSpace = yes ConstrainSwapSpace = no EnableControllers = no IgnoreSystemd = no IgnoreSystemdOnFailure = no MaxRAMPercent = 100.0% MaxSwapPercent = 100.0% MemorySwappiness = (null) MinRAMSpace = 30 MB MPI Plugins Configuration: PMIxCliTmpDirBase = (null) PMIxCollFence = (null) PMIxDebug = 0 PMIxDirectConn = yes PMIxDirectConnEarly = no PMIxDirectConnUCX = no PMIxDirectSameArch = no PMIxEnv = (null) PMIxFenceBarrier = no PMIxNetDevicesUCX = (null) PMIxTimeout = 300 PMIxTlsUCX = (null) the partition is configured like: PartitionName=test-partition AllowGroups=sysadmin,users AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=YES ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED Nodes=vserv-[275-277] PriorityJobFactor=1 PriorityTier=100 RootOnly=NO ReqResv=NO OverSubscribe=FORCE:1 OverTimeLimit=NONE PreemptMode=GANG,SUSPEND State=UP TotalCPUs=32 TotalNodes=4 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=32,mem=304933M,node=4,billing=32 Our QOS looks like this: | Name | Priority | GraceTime | Preempt | PreemptMode | Flags | UsageFactor | MaxTRESPU | MaxJobsPU | MaxSubmitPU |-----------|----------|-----------|-----------|-------------|-----------------|-------------|-----------------|-----------|----------- | normal | 50 | 00:00:00 | | cluster | | 1.000000 | | 20 | 50 | preempter | 1000 | 00:00:00 | preempted | gang,suspe+| | 1.000000 | | | | preempted | 0 | 00:00:00 | | gang,suspe+ | OverPartQOS | 1.000000 | cpu=100,node=10 | | I can provide more configs if needed. Do you guys see anything strange? Or any property to be set? This is the state of the queue: |JOBID | QOS | ST | TIME | NODELIST(REASON) | PARTITION | PRIORITY | | ---------------- ----------- --------- ----------- ---------------------- ----------------- ----------| | 3629 | preempted | PD | 0:00 | (Resources) | test-partition | 1 | | 3627 | preempted | R | 0:20 | vserv-276 | test-partition | 1 | | 3628 | preempted | R | 0:20 | vserv-277 | test-partition | 1 | | 3626 | preempted | R | 0:27 | vserv-275 | test-partition | 1 | | 3630 | preempter | PD | 0:00 | (Resources) | test-partition | 5000 | any advice is welcomed. Regards, Nischey Verma Nischey Verma

3 2

Issues with Slurm 23.11.1
by Fokke Dijkstra 15 Mar '24

15 Mar '24

Dear all, Since the upgrade from Slurm 22.05 to 23.11.1 we are having problems with the communication between the slurmctld and slurmd processes. We are running a cluster with 183 nodes and almost 19000 cores. Unfortunately some nodes are in a different network preventing full internode communication. A network topology and setting TopologyParam RouteTree have been used to make sure no slurmd communication happens between nodes on different networks. In the new Slurm version we see the following issues, which did not appear in 22.05: 1. slurmd processes acquire many network connections in CLOSE-WAIT (or CLOSE_WAIT depending on the tool used) causing the processes to hang, when trying to restart slurmd. When checking for CLOSE-WAIT processes we see the following behaviour: Recv-Q Send-Q Local Address:Port Peer Address:Port Process 1 0 10.5.2.40:6818 10.5.0.43:58572 users:(("slurmd",pid=1930095,fd=72),("slurmd",pid=1930067,fd=72)) 1 0 10.5.2.40:6818 10.5.0.43:58284 users:(("slurmd",pid=1930095,fd=8),("slurmd",pid=1930067,fd=8)) 1 0 10.5.2.40:6818 10.5.0.43:58186 users:(("slurmd",pid=1930095,fd=22),("slurmd",pid=1930067,fd=22)) 1 0 10.5.2.40:6818 10.5.0.43:58592 users:(("slurmd",pid=1930095,fd=76),("slurmd",pid=1930067,fd=76)) 1 0 10.5.2.40:6818 10.5.0.43:58338 users:(("slurmd",pid=1930095,fd=19),("slurmd",pid=1930067,fd=19)) 1 0 10.5.2.40:6818 10.5.0.43:58568 users:(("slurmd",pid=1930095,fd=68),("slurmd",pid=1930067,fd=68)) 1 0 10.5.2.40:6818 10.5.0.43:58472 users:(("slurmd",pid=1930095,fd=69),("slurmd",pid=1930067,fd=69)) 1 0 10.5.2.40:6818 10.5.0.43:58486 users:(("slurmd",pid=1930095,fd=38),("slurmd",pid=1930067,fd=38)) 1 0 10.5.2.40:6818 10.5.0.43:58316 users:(("slurmd",pid=1930095,fd=29),("slurmd",pid=1930067,fd=29)) The first IP address is that of the compute node, the second that of the node running slurmctld. The nodes can communicate using these IP addresses just fine. 2. slurmd cannot be properly restarted [2024-01-18T10:45:26.589] slurmd version 23.11.1 started [2024-01-18T10:45:26.593] error: Error binding slurm stream socket: Address already in use [2024-01-18T10:45:26.593] fatal: Unable to bind listen port (6818): Address already in use This is probably because of the processes being in CLOSE-WAIT, which can only be killed using signal -9. 3. We see jobs stuck in completing CG state, probably due to communication issues between slurmctld and slurmd. The slurmctld sends repeated kill requests but those do not seem to be acknowledged by the client. This happens more often in large job arrays, or generally when many jobs start at the same time. However, this could be just a biased observation (i.e., it is more noticeable on large job arrays because there are more jobs to fail in the first place). 4. Since the new version we also see messages like: [2024-01-17T09:58:48.589] error: Failed to kill program loading user environment [2024-01-17T09:58:48.590] error: Failed to load current user environment variables [2024-01-17T09:58:48.590] error: _get_user_env: Unable to get user's local environment, running only with passed environment The effect of this is that the users run with the wrong environment and can’t load the modules for the software that is needed by their jobs. This leads to many job failures. The issue appears to be somewhat similar to the one described at: https://bugs.schedmd.com/show_bug.cgi?id=18561 In that case the site downgraded the slurmd clients to 22.05 which got rid of the problems. We’ve now downgraded the slurmd on the compute nodes to 23.02.7 which also seems to be a workaround for the issue. Does anyone know of a better solution? Kind regards, Fokke Dijkstra -- Fokke Dijkstra <f.dijkstra(a)rug.nl> <f.dijkstra(a)rug.nl> Team High Performance Computing Center for Information Technology, University of Groningen Postbus 11044, 9700 CA Groningen, The Netherlands

2 3

Overhead of multiple concurrent job steps
by Mehta, Kshitij 15 Mar '24

15 Mar '24

Hello, We have a use case in which we need to launch multiple concurrently running MPI applications inside a job allocation. Most supercomputing facilities limit the number of concurrent job steps as they incur an overhead with the global Slurm scheduler. Some frameworks, such as the Flux framework from LLNL, claim to mitigate this issue by starting an instance of their own scheduler inside an allocation, which then acts as the resource manager for the compute nodes in the allocation. Out of curiosity, I was wondering if there is a fundamental reason behind having a single global scheduler that the srun launch commands must contact to launch job steps. Perhaps it was overkill to develop a ‘hierarhical’ design in which Slurm launches a local job daemon for every allocation that manages resources for that allocation? I would appreciate your insight in understanding more about Slurm’s core design. Thanks and regards, Kshitij Mehta Oak Ridge National Laboratory

1 0

Jobs being denied for GrpCpuLimit despite having enough resource
by Simon Andrews 14 Mar '24

14 Mar '24

Our cluster has developed a strange intermittent behaviour where jobs are being put into a pending state because they aren't passing the AssocGrpCpuLimit, even though the user submitting has enough cpus for the job to run. For example: $ squeue -o "%.6i %.9P %.8j %.8u %.2t %.10M %.7m %.7c %.20R" JOBID PARTITION NAME USER ST TIME MIN_MEM MIN_CPU NODELIST(REASON) 799 normal hostname andrewss PD 0:00 2G 5 (AssocGrpCpuLimit) ..so the job isn't running, and it's the only job in the queue, but: $ sacctmgr list associations part=normal user=andrewss format=Account,User,Partition,Share,GrpTRES Account User Partition Share GrpTRES ---------- ---------- ---------- --------- ------------- andrewss andrewss normal 1 cpu=5 That user has a limit of 5 CPUs so the job should run. The weird thing is that this effect is intermittent. A job can hang and the queue will stall for ages but will then suddenly start working and you can submit several jobs and they all work, until one fails again. The cluster has active nodes and plenty of resource: $ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST normal* up infinite 2 idle compute-0-[6-7] interactive up 1-12:00:00 3 idle compute-1-[0-1,3] The slurmctld log just says: [2024-03-14T16:21:41.275] _slurm_rpc_submit_batch_job: JobId=799 InitPrio=4294901720 usec=259 Whilst it's in this state I can run other jobs with core requests of up to 4 and they work, but not 5. It's like slurm is adding one CPU to the request and then denying it. I'm sure I'm missing something fundamental but would appreciate it if someone could point out what it is! Thanks Simon.

3 3

We're Live! Check out the new SchedMD.com now!
by Victoria Hobson 13 Mar '24

13 Mar '24

Our website has gone through some much needed change and we'd love for you to explore it! The new SchedMD.com is equipped with the latest information about Slurm, your favorite workload manager, and details about SchedMD services, support, and training offerings. Toggle through our Industries pages (https://www.schedmd.com/slurm-industries/) to learn more about how Slurm can service your specific site needs. Why Slurm? (https://www.schedmd.com/slurm/why-slurm/) gives you all the basics around our market-leading scheduler and SchedMD Services (https://www.schedmd.com/slurm-support/our-services/) addresses all the ways we can help you optimize your site. These new web pages also feature access to our Documentation Site, Bug Site, and Installation Guide. Browse our Events tab to see where we'll be when, and be sure to register for our Slurm User Group (SLUG) in Oslo, Norway this fall! (https://www.schedmd.com/about-schedmd/events/) SchedMD.com, your one stop shop for all things Slurm. Check it out now! -- Victoria Hobson SchedMD LLC Vice President of Marketing

2 1

Slurm Utilities
by Paul Edmon 13 Mar '24

13 Mar '24

Just wanted to share some slurm utilities that we've written at Harvard FASRC that maybe useful to the community. seff-account: https://github.com/fasrc/seff-account Creates job statistics summaries for users and accounts similar to what seff and seff-array does. showq: https://github.com/fasrc/slurm_showq A slurm version of the Moab showq command lsload: https://github.com/fasrc/lsload A slurm version of the LSF lsload command scalc: https://github.com/fasrc/scalc A calculator for various fairshare related things spart: https://github.com/fasrc/spart A simplified output for slurm partition information stdg: https://github.com/fasrc/stdg Slurm test deck generator prometheus-slurm-exporter: https://github.com/fasrc/prometheus-slurm-exporter Slurm exporters for prometheus Hopefully people find these useful. Pull requests are always appreciated. -Paul Edmon-

1 0

Re: SLURM in K8s, any advice?
by LEAVY Alan 13 Mar '24

13 Mar '24

I'm a little late to this party but would love to establish contact with others using slurm in Kubernetes. I recently joined a research institute in Vienna (IIASA) and I'm getting to grips with slurm and Kubernetes (my previous role was data engineering / fintech). My current setup sounds like what Urban described in this thread, back in Nov 22. It has some rough edges though. Right now, I'm trying to upgrade to slurm-23.11.4 in Ubuntu 23.10 containers. I'm having trouble with the cgroup/v2 plugin. Are you still using slurm on K8s Urban? How did your installation work out Hans? Would either of you be willing to share your experiences? Regards, Alan.

4 3

Shards distribution over multiple GPUs
by Francois Broquedis 12 Mar '24

12 Mar '24

Hi all, We're trying to enable sharding on our compute cluster. On this cluster: - ensicompute-1 comes with 1 NVIDIA V100 GPU ; - ensicompute-13 comes with 3 NVIDIA A40 GPUs ; - all other nodes (for now, ensicompute-11 and ensicompute-12, but several others will come) come with 3 NVIDIA RTX 6000 GPUs. To enable sharding, I followed these steps: 1. [slurm.conf] Add "shard" to GresTypes ; 2. [slurm.conf] Add "shard:N" to Gres for each node. For testing purposes, I have set N to 9, so each GPU can execute up to 3 jobs concurrently: NodeName=ensicompute-[11-12] Gres=gpu:Quadro:3,shard:9 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht 3. [gres.conf] Declare the shards after the definition of the GPUs GRES. For step 3, I tried different things, leading to different outcomes: a. Define a global number of shards, for the entire host: Name=shard Count=9 ==> This way, sharding seems to work ok, but all the jobs are executed on GPU#0. If running 12 jobs for example, 9 of them are assigned to GPU#0 and start executing, while 3 of them remain in a pending state. No job is assigned to GPU#1 or GPU#2. b. Define a per-GPU number of shards, associated to the device file representing the GPU: Name=shard Count=3 File=/dev/nvidia0 Name=shard Count=3 File=/dev/nvidia1 Name=shard Count=3 File=/dev/nvidia2 ==> In this case, the slurmd service fails to start on the compute node. The error message found in /var/log/slurmd.log is "fatal: Invalid GRES record for shard, count does not match File value". c. Don't define anything about shards in gres.conf. ==> Same behavior than in a.: all jobs are executed on GPU#0. I attach to this message the full content of the slurm.conf and gres.conf files. What is the proper way to configure sharding in a cluster with several GPUs per node? Is there a way to specify how many shards should be allocated to each GPU? Cheers, François === slurm.conf === # slurm.conf file generated by configurator.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # ClusterName=ensimag SlurmctldHost=nash ProctrackType=proctrack/cgroup SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm #SlurmdUser=root StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/affinity,task/cgroup ReturnToService=2 # # # TIMERS InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres # # # LOGGING AND ACCOUNTING JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log # # # COMPUTE NODES GresTypes=gpu,shard NodeName=ensicompute-1 Gres=gpu:Tesla:1,shard:3 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht NodeName=ensicompute-13 Gres=gpu:A40:3,shard:9 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht NodeName=ensicompute-[11-12] Gres=gpu:Quadro:3,shard:9 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht PartitionName=compute Nodes=ALL Default=YES MaxTime=INFINITE State=UP === gres.conf === AutoDetect=off # ensicompute-1 NodeName=ensicompute-1 Name=gpu Type=Tesla File=/dev/nvidia0 NodeName=ensicompute-1 Name=shard Count=3 File=/dev/nvidia0 # ensicompute-11 NodeName=ensicompute-11 Name=gpu Type=Quadro File=/dev/nvidia0 NodeName=ensicompute-11 Name=gpu Type=Quadro File=/dev/nvidia1 NodeName=ensicompute-11 Name=gpu Type=Quadro File=/dev/nvidia2 NodeName=ensicompute-11 Name=shard Count=3 File=/dev/nvidia0 NodeName=ensicompute-11 Name=shard Count=3 File=/dev/nvidia1 NodeName=ensicompute-11 Name=shard Count=3 File=/dev/nvidia2 # ensicompute-12 NodeName=ensicompute-12 Name=gpu Type=Quadro File=/dev/nvidia0 NodeName=ensicompute-12 Name=gpu Type=Quadro File=/dev/nvidia1 NodeName=ensicompute-12 Name=gpu Type=Quadro File=/dev/nvidia2 NodeName=ensicompute-12 Name=shard Count=3 File=/dev/nvidia0 NodeName=ensicompute-12 Name=shard Count=3 File=/dev/nvidia1 NodeName=ensicompute-12 Name=shard Count=3 File=/dev/nvidia2 # ensicompute-13 NodeName=ensicompute-13 Name=gpu Type=A40 File=/dev/nvidia0 NodeName=ensicompute-13 Name=gpu Type=A40 File=/dev/nvidia1 NodeName=ensicompute-13 Name=gpu Type=A40 File=/dev/nvidia2 NodeName=ensicompute-13 Name=shard Count=3 File=/dev/nvidia0 NodeName=ensicompute-13 Name=shard Count=3 File=/dev/nvidia1 NodeName=ensicompute-13 Name=shard Count=3 File=/dev/nvidia2 -- François Broquedis, Ingénieur Service Informatique Grenoble INP - Ensimag, bureau E208 681 rue de la Passerelle BP 72, 38402 Saint Martin d'Hères CEDEX Tél.: +33 (0)4 76 82 72 78

1 0

Re: user_name attribute missing in job response from slurm rest api method jobs
by Acartürk, Ayhan 11 Mar '24

11 Mar '24

Sorry... my previous message should have been a reply to the post by Filip Holka from March, 25th 2022: https://lists.schedmd.com/pipermail/slurm-users/2022-March/008543.html Best wishes, Ayhan Acartürk

1 0

Re: user_name attribute missing in job response from slurm rest api method jobs
by Acartürk, Ayhan 11 Mar '24

11 Mar '24

Hello together, we have the same problem here. All jobs delivered by the api endpoint /slurm/v0.0.36/jobs have a user_id attached but no user_name. Currently, we solve this issue by an extra call for each job_id to the api endpoint /slurmdb/v.0.0.36/job/{job_id}. Any idea where to search for a possible solution is highly appreciated. Best wishes, Ayhan Acartürk

1 0

Problem building slurm with PMIx
by Patrick Begou 11 Mar '24

11 Mar '24

Hi ! I manage a small CentOS8 cluster using slurm slurm-20.11.7-1 and OpenMPI built from sources. - I know this OS is not maintained any more and I need to negotiate downtime to reinstall - I know Slurm 20.11.7 has security issue (I've built it from source some years ago with rpmbuild -ta --with mysql --with hwloc slurm-20.11.7.tar.bz) and I should update. All was running fine until I add a GPU Node and Nvidia sdk. This SDK provides an openMPI3 implementation GPU aware but I'm unable to launch an intranode parallel job with it using srun: -------------------------------------------------------------------------- The application appears to have been direct launched using "srun", but OMPI was not built with SLURM's PMI support and therefore cannot execute. There are several options for building PMI support under SLURM, depending upon the SLURM version you are using: version 16.05 or later: you can use SLURM's PMIx support. This requires that you configure and build SLURM --with-pmix. Versions earlier than 16.05: you must use either SLURM's PMI-1 or PMI-2 support. SLURM builds PMI-1 by default, or you can manually install PMI-2. You must then build Open MPI using --with-pmi pointing to the SLURM PMI library location. Please configure as appropriate and try again. -------------------------------------------------------------------------- I check with "srun --mpi=list" and got no pmx. srun: MPI types are... srun: pmi2 srun: cray_shasta srun: none So I decide to build the rpms from slurm-20.11.9.tar.bz2 as I had done previously for 20.11.7 and update. I've first installed pmix-2.1.1-1 from src as I had no pmix-devel rpm in my local CentOS8 repo: rpm --rebuild pmix-2.1.1-1.el8.src.rpm dnf install pmix-devel-2.1.1-1.el8.x86_64.rpm pmix-2.1.1-1.el8.x86_64.rpm Then build Slurm from slurm-20.11.9.tar.bz2 (just changing python3 to python38 in the spec file) rpmbuild -ta --with mysql --with hwloc --with pmix slurm-20.11.9.tar.bz2 And the try to install these package on the GPU node dnf install slurm-slurmd-20.11.9-1.el8.x86_64.rpm slurm-20.11.9-1.el8.x86_64.rpm slurm-devel-20.11.9-1.el8.x86_64.rpm slurm-libpmi-20.11.9-1.el8.x86_64.rpm But I get this strange error: Error: Problem: conflicting requests - nothing provides pmix = 20.11.9 needed by slurm-slurmd-20.11.9-1.el8.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) Why this request on PMIX with the slurm version number ? Am I wrong somewhere ? Thanks for your help Patrick

2 1

FairShare value is always 0.
by Zacarias Benta 07 Mar '24

07 Mar '24

Hi guys, We've just setup our new cluster and are facing some issues regading fairshare calculation. Our slurm directive regarding priority calculation are defines as follows: PriorityType=priority/multifactor PriorityFlags=MAX_TRES PriorityDecayHalfLife=14-0 PriorityFavorSmall=NO PriorityMaxAge=14-0 PriorityWeightAge=1000 PriorityWeightJobSize=1000 PriorityWeightPartition=10000000 PriorityWeightQOS=10000000 PriorityWeightTRES=CPU=2000,Mem=4000 PriorityWeightFairshare=100000 The partition we are submitinh out jobs to is setup as follows: PartitionName=mypartPriority=1000TRESBillingWeights="CPU=1.0,Mem=0.25G"Default=YESMaxTime=96:0:0DefMemPerCPU=5333Nodes=node[001-036] MaxNodes=20 Whenever we take a look at the fairshare value using sshare -l we see the following output: AccountUserRawSharesNormSharesRawUsageNormUsageEffectvUsageFairShareLevelFSGrpTRESMinsTRESRunMins ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- root10.0000002687245970.0000000.000000cpu=1098201,mem=5856709132,en+ rootroot10.10000000.0000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group110.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group210.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group310.1000002687245970.0000000.000000cpu=1098201,mem=5856709132,en+ group410.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group510.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group610.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group710.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group810.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group910.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ We think it is really weird that the FairShare value is 0 for the root account and "NULL" for all other groups, even the one who had the greatest raw usage. While taking a look at the data for our users we see the following: AccountUserRawSharesNormSharesRawUsageEffectvUsageFairShare ------------------------------------------------------------------------------------- root10.0000002689837210.000000 rootroot10.10000000.0000000.000000 group310.1000002689837210.000000 group3user110.090909121093740.0000000.000000 group3user210.09090900.0000000.000000 group3user310.09090900.0000000.000000 group3user410.09090900.0000000.000000 group3user510.09090900.0000000.000000 group3user610.09090900.0000000.000000 group3user710.09090900.0000000.000000 group3user810.0909092088245970.0000000.000000 group3user910.09090900.0000000.000000 group3user1010.09090900.0000000.000000 group3user1110.090909480497500.0000000.000000 group410.10000000.000000 group4user1310.0000004994520.0000000.000000 group510.10000000.000000 group5user1410.00000015396030.0000000.000000 This is a weird behavior, since user1, user8, user11, user13 and user14 are the ones who have more RawUsage and the FairShare value is the same for all of them, including the users that have no yet submited any job. We also noticed that in the slurmctld log there is the fillowing error message that appears with some regularity [2024-03-07T16:38:13.260] error: _append_list_to_array: unable to append NULL list to assoc list. [2024-03-07T16:38:13.260] error: _calc_tree_fs: unable to calculate fairshare on empty tree The error above looks like it is coming from: https://github.com/SchedMD/slurm/blob/b11bf689b270f1f5dfe4b0cd54c4fa84b4af3… Are we missing any setting on slurm.conf? This is kind of strange, because we have another cluster with pretty much the same configuration and the FairShare is calculated without any problems. Any help would be appreciated. -- Cumprimentos / Best Regards, Zacarias Benta LIP/INCD @ UMINHO ---------------------------------------------- / Use linux, and may the source be with you. / ---------------------------------------------- \ __ -=(o '. '.-.\ /| \\ '| || _\_):,_

1 0

Slurm management of dual-node server trays?
by Ole Holm Nielsen 07 Mar '24

07 Mar '24

We're in the process of installing some racks with Lenovo SD665 V3 [1] water-cooled servers. A Lenovo DW612S chassis contains 6 1U trays with 2 SD665 V3 servers mounted side-by-side in each tray. Lenovo delivers SD665 V3 servers including water-cooled NVIDIA InfiniBand "SharedIO" adapters [2] so that one node is the Primary including a PCIe adapter, and the other is Auxiliary with just a cable to the Primary's adapter. Obviously, servicing 2 "Siamese twin" Slurm nodes requires a bit of care and planning. What is worse is that when the Primary node is rebooted or powered down, the Auxiliary node will lose its Infiniband connection and may have a PCIe fault or an NMI as documented in [3]. And when nodes are powered up, the Primary must have completed POST before the Auxiliary gets started. I wonder how to best deal with power failures? It seems that when Slurm jobs are running on Auxiliary nodes, these jobs are going to crash when the possibly unrelated Primary node goes down. This looks like a pretty bad system design on the part of Lenovo :-( The goal was apparently to same some money on IB adapters and having fewer IB cables. Question: Do any Slurm sites out there already have experiences with Lenovo "Siamese twin" nodes with SharedIO IB? Have you developed some operational strategies, for example dealing with node pairs as a single entity for job scheduling? Thanks for sharing any ideas and insights! Ole [1] https://lenovopress.lenovo.com/lp1612-lenovo-thinksystem-sd665-v3-server [2] https://lenovopress.lenovo.com/lp1693-thinksystem-nvidia-connectx-7-ndr200-… [3] https://support.lenovo.com/us/en/solutions/ht510888-thinksystem-sd650-and-c… -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

3 3

Re: Lua script
by Paul Raines 06 Mar '24

06 Mar '24

Alternativelly consider setting EnforcePartLimits=ALL in slurm.conf -- Paul Raines (http://help.nmr.mgh.harvard.edu) The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

Re: Lua script
by Gestió Servidors 06 Mar '24

06 Mar '24

And how can I reject the job inside the lua script?

2 1

Lua script
by Gestió Servidors 06 Mar '24

06 Mar '24

Hello, I'm writing a small lua script that for modify "TimeLimit" of a submited job if user has configured a TimeLimit bigger that configured in the partition. So, is TimeLimit for partition is, for example, 4 hours (04:00:00) and user submit his/her job with a TimeLimit of 5 hours, lua script modify this submit and forces "TimeLimit" to 4 hours for avoiding that job remains at queue in state "pending" with reason "Partition Time Limit...". I have written these lines: function slurm_job_submit(job_desc, part_list, submit_uid) --[[ my id is 1008; for testing purposes, lua only applies to me --]] if (job_desc.user_id == 1008) then slurm.log_info("Submit job by me") --[[ I check partition --]] if (job_desc['partition'] == "nodo.q") then --[[ if (job_desc['time_limit'] > 14400) job_desc['time_limit'] = 14400 end end end return slurm.SUCCESS end However, if I submit a job with TimeLimit of 5 hours, lua script doesn't modify submit and job remains "pending"... What am I doing wrong? Thanks.

2 1

Slurm billback and sreport
by Chip Seraphine 05 Mar '24

05 Mar '24

Hello, I am attempting to implement a billback model and finding myself stymied by the way that sreport handles job arrays. Basically, when a user submits a large array, their usage includes time that jobs in the back of the array spend waiting their turn. (My #1 user in “sreport user topusage” shows more “used” cpu*minutes than the cluster physically _has_ during that interval.) However, jobs that are idle pending resources are simply regarded as pending; as a result, a “polite” user who submits an array of 1000 jobs running N at a time is penalized over a user who just dumps 1000 loose jobs into the queue. This incentives my users to do exactly what I do not want! Has anyone tried to bill their users based on the results of sreport? If so, how did you work around this problem? What did you use to determine the # of CPU*Minutes that a user actually allocated on during a given interval? -- Chip Seraphine Grid Operations For support please use help-grid in email or slack. This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

3 3

Is SWAP memory mandatory for SLURM
by John Joseph 04 Mar '24

04 Mar '24

Dear All, Good morning I do have a 4 node SLURM instance up and running. Like to know if I disable the SWAP memory, will it effect the SLURM performance Is SWAP a mandatory requirement, I have each node more RAM, if my phsicall RAM is more, is there any need for the SWAP thanks Joseph John

4 3

What is "eligible time"
by Oren Shani 04 Mar '24

04 Mar '24

Hi All, Can someone shed some light on the exact meaning of the "eligible time" attribute in the slurm accounter, or to be more exact on the time between when the jobs becomes "eligible" to the time it actually starts to run? I understand that "eligible time" refers to the time in which the job becomes a candidate for running and that one case in which it will still not run, is that it waits for resources to become available. But are there any other conditions for a job to keep waiting at this point? For example, is fair-share scheduling considered only at this point? Many thanks, Oren

1 0

Question about IB and Ethernet networks
by Dan Healy 04 Mar '24

04 Mar '24

Hi Fellow Slurm Users, This question is not slurm-specific, but it might develop into that. My question relates to understanding how *typical* HPCs are designed in terms of networking. To start, is it typical for there to be a high speed Ethernet *and* Infiniband networks (meaning separate switches, NICs)? I know you can easily setup IP over IB, but is IB usually fully reserved for MPI messages? I’m tempted to spec all new HPCs with only a high speed (200Gbps) IB network, and use IPoIB for all slurm comms with compute nodes. I plan on using BeeGFS for the file system with RDMA. Just looking for some feedback, please. Is this OK? Is there a better way? If yes, please share why it’s better. Thanks, Daniel Healy

6 6

How to get usage data for a QOS
by thomas.hartmann＠th-ht.de 01 Mar '24

01 Mar '24

Hi, so, I figured out that I can give some users priority access for a specific amount of TRES by creating a qos with the GrpTRESMins property and the DenyOnLimit,NoDecay flags. This works nicely. However, I would like to know, how much of this has already been consumed and I have not yet found a way to do this. Like: How can I get the amount of TRES/TRES minutes consumed for a certain QOS? Thanks a lot! Thomas

3 2

MinTRES in QoS and power saving
by Stefan Walter 01 Mar '24

01 Mar '24

I am running a GPU cluster where nodes are mostly off to save on electricity. I have run into the problem that if I set 'MinTRES=gres/gpu=1' in the QoS for user-account associations, waking up nodes on-demand stops working for these users. Jobs are allocated on all running nodes but if a user starts a job that would require a node to be woken up, it remains pending with 'QOSMinGRES'. In the verbose logs, such a job will show this: debug3: TRES Weight: cpu = 1.000000 * 0.000000 = 0.000000 debug3: TRES Weight: mem = 6000.000000 * 0.000000 = 0.000000 debug3: TRES Weight: energy = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: node = 1.000000 * 0.000000 = 0.000000 debug3: TRES Weight: fs/disk = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: vmem = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: pages = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: gres/gpu = 0.000000 * 1.000000 = 0.000000 debug3: TRES Weight: gres/gpu:nvidia_geforce_gtx_1080ti = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: gres/gpumem = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: gres/gpuutil = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weighted: SUM(TRES) = 0.000000 debug2: JobId=4256 is being held, QOS local818-test min tres(gres/gpu) per job request 0 exceeds min tres limit 1 The job was started with '-G 1' or '--gres=gpu:1' but notably the TRES value for 'gres/gpu' becomes 0. For jobs that fit on the running nodes the TRES weight remains correct: debug3: TRES Weight: gres/gpu = 1.000000 * 1.000000 = 1.000000 The quick fix was to unset MinTRES for all QoS but I still would force user in some accounts to use GPUs because that is what is billed for them. Is this by design? I was not successful to find any hint that this is expected behavior. Thanks, Stefan

1 0

pty jobs are killed when another job on the same node terminates
by Thomas Hartmann 01 Mar '24

01 Mar '24

Hi, when I start an interactive job like this: srun --pty --mem=3G -c2 bash And then I schedule and run other jobs (can be interactive or non interactive) and one of these jobs that runs on the same node terminates, the interactive job gets killed with this message: srun: error: node01.abc.at: task 0: Killed I attached our slurm config. Does anybody have an idea what is going on here or where I could look to debug? I'm quite new to slurm, so I don't know all the places to look... Thanks a lot in advance! Thomas

4 3