- slurm-users - lists.schedmd.com

Spread a multistep job across clusters
by Di Bernardini, Fabio 28 Aug '24

28 Aug '24

Hi everyone, for accounting reasons, I need to create only one job across two or more federated clusters with two or more srun steps. I'm trying with hetjobs but it's not clear to me from the documentation (https://slurm.schedmd.com/heterogeneous_jobs.html) if this is possible and how to do it. I'm trying with this script, but the steps are executed on only the first cluster. Can you tell me if there is a mistake in the hetjob or if it has to be done in another way? #!/bin/bash #SBATCH hetjob #SBATCH --clusters=cluster2 srun -v --het-group=0 hostname #SBATCH hetjob #SBATCH --clusters=cluster3 srun -v --het-group=1 hostname NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico

3 2

Slurm versions 24.05.3 and 23.11.10 are now available
by Marshall Garey 27 Aug '24

27 Aug '24

We are pleased to announce the availability of Slurm versions 24.05.3 and 23.11.10. Version 24.05.3 fixes a potential database problem when deleting a qos. This bug only existed in 24.05. Both versions have fixes for jobs potentially being stuck when using cloud nodes when some nodes are powered down, a regression in 23.11.9 and 24.05.2 that caused sattach to crash, and some other minor issues. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.05.3 > ========================== > -- data_parser/v0.0.40 - Added field descriptions > -- slurmrestd - Avoid creating new slurmdbd connection per request to > '* /slurm/slurmctld/*/*' endpoints. > -- Fix compilation issue with switch/hpe_slingshot plugin. > -- Fix gres per task allocation with threads-per-core. > -- data_parser/v0.0.41 - Added field descriptions > -- slurmrestd - Change back generated OpenAPI schema for > `DELETE /slurm/v0.0.40/jobs/` to RequestBody instead of using parameters > for request. slurmrestd will continue accept endpoint requests via > RequestBody or HTTP query. > -- topology/tree - Fix issues with switch distance optimization. > -- Fix potential segfault of secondary slurmctld when falling back to the > primary when running with a JobComp plugin. > -- Enable --json/--yaml=v0.0.39 options on client commands to dump data using > data_parser/v0.0.39 instead or outputting nothing. > -- switch/hpe_slingshot - Fix issue that could result in a 0 length state file. > -- Fix unnecessary message protocol downgrade for unregistered nodes. > -- Fix unnecessarily packing alias addrs when terminating jobs with a mix of > non-cloud/dynamic nodes and powered down cloud/dynamic nodes. > -- accounting_storage/mysql - Fix issue when deleting a qos that could remove > too many commas from the qos and/or delta_qos fields of the assoc table. > -- slurmctld - Fix memory leak when using RestrictedCoresPerGPU. > -- Fix allowing access to reservations without MaxStartDelay set. > -- Fix regression introduced in 24.05.0rc1 breaking srun --send-libs parsing. > -- Fix slurmd vsize memory leak when using job submission/allocation commands > that implicitly or explicitly use --get-user-env. > -- slurmd - Fix node going into invalid state when using CPUSpecList and > setting CPUs to the # of cores on a multithreaded node > -- Fix reboot asap nodes being considered in backfill after a restart. > -- Fix --clusters/-M queries for clusters outside of a federation when > fed_display is configured. > -- Fix scontrol allowing updating job with bad cpus-per-task value. > -- sattach - Fix regression from 24.05.2 security fix leading to crash. > -- mpi/pmix - Fix assertion when built under --enable-debug. > * Changes in Slurm 23.11.10 > =========================== > -- switch/hpe_slingshot - Fix issue that could result in a 0 length state file. > -- Fix unnecessary message protocol downgrade for unregistered nodes. > -- Fix unnecessarily packing alias addrs when terminating jobs with a mix of > non-cloud/dynamic nodes and powered down cloud/dynamic nodes. > -- Fix allowing access to reservations without MaxStartDelay set. > -- Fix scontrol allowing updating job with bad cpus-per-task value. > -- sattach - Fix regression from 23.11.9 security fix leading to crash.

1 0

Slurm management of Lenovo SD665 V3 dual-server trays?
by Ole Holm Nielsen 26 Aug '24

26 Aug '24

We're experimenting with ways to manage our new racks of Lenovo SD665 V3 dual-server trays with Direct Water Cooling (further information is in our Wiki page https://wiki.fysik.dtu.dk/ITwiki/Lenovo_SD665_V3/ ) Management problems arise because 2 servers share a tray with common power and water cooling. This wouldn't be so bad if it weren't for Lenovo's NVIDIA/Mellanox SharedIO Infiniband adapters, where the left-hand node's IB adapter is a client of the right-hand node's adapter. So we can't reboot or power down the right-hand node without killing any MPI jobs that happen to be using the left-hand node. My question is if other Slurm sites owning Lenovo dual-server trays with SharedIO Infiniband adapters have developed some clever ways of handling such node pairs a single entity somehow? Is there anything we should configure on the Slurm side to make such nodes easier to manage? Thanks for sharing any insights, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 5

How to select a container runtime system?
by wdennis＠nec-labs.com 23 Aug '24

23 Aug '24

We are getting a few calls to support container workloads on our Slurm cluster; I want to support these user's usecases, so am looking into it now. The problem for me is, I'm not super-familiar with container runtimes excepting (regular rootful) Docker... I see that any Slurm-compatible runtime must be "rootless", so that takes out the integration with regular Docker. How does one choose between the many options (from what I gather, Apptainer/Singularity, Podman, and rootless Docker seem to be the most widely-used.) We also have to support GPUs in these containers, but I assume that any of these can support that. Can anyone tell me what the most popular Slurm-compatible container runtime may be, and plusses/minuses of the three I mentioned above? Thanks!

2 1

Slurmdbd purge and reported downtime
by Davide DelVento 23 Aug '24

23 Aug '24

I am confused by the reported amount of Down and PLND Down by sreport. According to it, our cluster would have had a significant amount of downtime, which I know didn't happen (or, according to the documentation "time that slurmctld was not responding", see https://slurm.schedmd.com/sreport.html) Could it be my purge settings causing this problem? How can I check (maybe in some logs, maybe in the future) if actually slurmctld was not responding? The expected long-term numbers should be less than the ones reported for last month when we had an issue with a few nodes.... Thanks! [davide@login ~]$ grep Purge /opt/slurm/slurmdbd.conf #JobPurge=12 #StepPurge=1 PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/22 -------------------------------------------------------------------------------- Cluster Utilization 2022-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- --------------- --------------- -------------- ---------------- --------- ---------------- cluster cpu 19.50% 12.07% 3.92% 64.36% 0.15% 100.03% cluster mem 16.13% 13.17% 4.56% 66.13% 0.00% 99.99% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/23 -------------------------------------------------------------------------------- Cluster Utilization 2023-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- --------------- --------------- -------------- --------------- --------- ---------------- cluster cpu 28.74% 18.80% 6.44% 45.77% 0.24% 100.02% cluster mem 22.52% 20.54% 7.38% 49.55% 0.00% 99.98% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- -------------- --------------- -------------- --------------- -------- --------------- cluster cpu 29.92% 24.88% 17.73% 27.45% 0.02% 100.00% cluster mem 20.07% 28.60% 19.57% 31.76% 0.00% 100.00% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=8/8/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-08-08T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Planned Reported --------- -------------- ------------- ------------ -------- -------------- -------- -------------- cluster cpu 15.96% 2.53% 0.00% 81.51% 0.00% 100.00% cluster mem 9.18% 2.22% 0.00% 88.60% 0.00% 100.00% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=7/7/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-07-07T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Planned Reported --------- -------------- -------------- ------------- -------- -------------- -------- -------------- cluster cpu 27.07% 2.57% 0.00% 70.34% 0.02% 100.00% cluster mem 17.35% 2.26% 0.00% 80.40% 0.00% 100.00%

2 2

trying to figure out how to troubleshoot cloud node resume/suspend
by Alex Chekholko 23 Aug '24

23 Aug '24

Hi all, I have a cloud cluster running in GCP that seems to have gotten stuck in a state where the slurmctld will not start/stop compute nodes, it just sits there with thousands of jobs in the queue and only a few compute nodes up and running (out of thousands). I can try to kick it by setting node states to drain/resume/power_up/power_down, whatever, nothing seems to happen. In slurmctld.log I get stuff like this: 2024-08-22 16:44:31,761 INFO: created 1 instances: nodes=hpcmarjola-computenodeset-384 but then in resume.log there is no corresponding entry One hypothesis I have is that it's taking like 30+mins to spin up a new node, but I know GCP doesn't take that long to create the VM, so how can I troubleshoot more what the slurm resume script is doing? e.g. I saw in the log: [2024-08-22T16:39:57.151] powering up node hpcmarjola-computenodeset-7 [2024-08-22T16:59:10.371] Node hpcmarjola-computenodeset-7 now responding I tried setting SlurmctldDebug = verbose #instead of info, so it prints power / resume status messages to the log And I see things like this: [2024-08-23T16:40:29.006] node hpcmarjola-computenodeset-1068 not resumed by ResumeTimeout(600) - marking down and power_save [2024-08-23T16:40:29.006] requeue job JobId=184535_503(185038) due to failure of node hpcmarjola-computenodeset-1068 [2024-08-23T16:40:29.006] Requeuing JobId=184535_503(185038) [2024-08-23T16:40:29.007] POWER: power_save: waking nodes hpcmarjola-computenodeset-[2154-2156] [2024-08-23T16:40:29.007] POWER: power_save: handle failed nodes hpcmarjola-computenodeset-[1065-1068] I'll try bumping this value but that seems crazy high: ResumeTimeout=2400 slurm 23.11.7 I read through these pages: https://slurm.schedmd.com/power_save.html my config files (generated by gcp hpc toolkit scripts): topology.conf: SwitchName=nodeset-root Switches=computenodeset SwitchName=computenodeset Nodes=hpcmarjola-computenodeset-[0-4999] cloud.conf: PrivateData=cloud LaunchParameters=enable_nss_slurm,use_interactive_step SlurmctldParameters=cloud_dns,enable_configless,idle_on_node_suspend SchedulerParameters=bf_continue,salloc_wait_nodes,ignore_prefer_validation SuspendProgram=/slurm/scripts/suspend.py ResumeProgram=/slurm/scripts/resume.py ResumeFailProgram=/slurm/scripts/suspend.py ResumeRate=2 ResumeTimeout=600 SuspendRate=2 SuspendTimeout=600 TreeWidth=16 TopologyPlugin=topology/tree NodeSet=x-login Feature=x-login PartitionName=x-login Nodes=x-login State=UP DefMemPerCPU=1 Hidden=YES RootOnly=YES NodeName=DEFAULT State=UNKNOWN RealMemory=126832 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=1 CPUs=8 NodeName=hpcmarjola-computenodeset-[0-4999] State=CLOUD NodeSet=computenodeset Nodes=hpcmarjola-computenodeset-[0-4999] PartitionName=compute Nodes=computenodeset State=UP DefMemPerCPU=15854 SuspendTime=600 Oversubscribe=Exclusive PowerDownOnIdle=NO Default=YES ResumeTimeout=600 SuspendTimeout=600 SuspendExcParts=x-login slurm.conf: # slurm.conf # https://slurm.schedmd.com/slurm.conf.html # https://slurm.schedmd.com/configurator.html ProctrackType=proctrack/cgroup SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmdPidFile=/var/run/slurm/slurmd.pid TaskPlugin=task/affinity,task/cgroup MaxNodeCount=64000 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory # # # LOGGING AND ACCOUNTING AccountingStoreFlags=job_comment JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=info SlurmdDebug=info DebugFlags=Power # # # TIMERS MessageTimeout=60 ################################################################################ # vvvvv WARNING: DO NOT MODIFY SECTION BELOW vvvvv # ################################################################################ SlurmctldHost=hpcmarjola-controller(internal-cloud-dns-redacted) AuthType=auth/munge AuthInfo=cred_expire=120 AuthAltTypes=auth/jwt CredType=cred/munge MpiDefault=none ReturnToService=2 SlurmctldPort=6820-6830 SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurm # # # LOGGING AND ACCOUNTING AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=hpcmarjola-controller ClusterName=hpcmarjola SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd-%n.log # # # GENERATED CLOUD CONFIGURATIONS include cloud.conf ################################################################################ # ^^^^^ WARNING: DO NOT MODIFY SECTION ABOVE ^^^^^ # ################################################################################ MaxArraySize = 200001 MaxJobCount = 200001

1 0

Manually initiating a DB purge
by Sid Young 22 Aug '24

22 Aug '24

G'Day all, I have 3 years worth of job records in the slurm DB and we do not have any need to actually track anything at this stage, I would like to keep 12months worth of jobs so I need to purge 2 years worth at some point.. Is there a command to issue via scontrol to kick off a SlurmPurge using the values in the slurmdbd.conf file (currently these are 1000days, so I can see a few dozen records disappear before I shorten it to 365days in incremental steps). Sid

2 4

active reservation lost when increasing the duration
by Danny Marc Rotscher 22 Aug '24

22 Aug '24

Dear All, we are running Slurm 23.11.8 and I loose an active reservation and the jobs running in that reservation, when I tried to increase the duration of that reservation. I only want to tell you, that you may not run into the same problem. Kind regards, Danny Rotscher

1 0

Slurm hanging behavior
by Richard Yang 20 Aug '24

20 Aug '24

Hello Slurm community, We are using slurm as the system to deploy training jobs on a large gpu cluster, but encounter a strange behavior. As new comers, we wonder if this is a known behavior. Below is some more info: * We are running a relatively older version 22.0.5 * At relatively higher load, we encountered hanging. It is particularly puzzling in the following sense: assume we have nodelist1 with 6 hosts and nodelist2 with 7 hosts. We run simple ‘hostname’. Deploying on nodelist1 alone or nodelusr2 alone will be fine, but with all 13 hosts, the debug messages show that the execution hang after showing that the last task done. It then hangs for exactly 180 seconds. Does anyone know the potential issue? We sure be happy to post more config details or debug messages. Thank you so much! Richard

1 0

Access to --constraint=<arg> in Lua cli_filter?
by Kevin Buckley 20 Aug '24

20 Aug '24

If I supply a --constraint=<arg> option to an sbatch/salloc/srun, does the arg appear inside any object that a Lua CLI Filter could access? I've tried this basic check if is_unset(options['constraint']) then slurm_errorf('constraint is unset ') end and seen that that object is, indeed, unset. Kevin

2 2

2025

2024

slurm-users ----- 2025 ----- May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users