August 2024 - slurm-users - lists.schedmd.com

Why AllowAccounts not work in slurm-23.11.6
by daijiangkuicgo＠gmail.com 30 Oct '24

30 Oct '24

I have set AllowAccounts=sunlabc5hpc,root, but it doesn’t seem to work. User c010637 is not part of the sunlabc5hpc account but is still able to use the sunlabc5hpc partition. I have tried setting EnforcePartLimits to ALL, ANY, and NO, but none of these options resolved the issue. [c010637@sl-login ~]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cpu* up infinite 3 mix sl-c[0035,0042-0043] cpu* up infinite 1 idle sl-c0036 gpu up infinite 3 idle sl-c[0045-0047] sunlabc5hpc up infinite 1 idle sl-c0048 [c010637@sl-login ~]$ scontrol show partition sunlabc5hpc PartitionName=sunlabc5hpc AllowGroups=ALL AllowAccounts=sunlabc5hpc,root AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED Nodes=sl-c0048 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=256 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=256,mem=515000M,node=1,billing=256,gres/gpu=8 [c010637@sl-login ~]$ sacctmgr list assoc format=cluster,user,account%20,qos user=$USER Cluster User Account QOS ---------- ---------- -------------------- -------------------- snowhpc c010637 c010637_bank normal [c010637@sl-login ~]$ sacctmgr list account sunlabc5hpc Account Descr Org ---------- -------------------- -------------------- sunlabc5h+ sunlabc5hpc sunlabc5hpc [c010637@sl-login ~]$ sacctmgr show assoc where Account=sunlabc5hpc format=User,Account User Account ---------- ---------- sunlabc5h+ c010751 sunlabc5h+ snowdai sunlabc5h+

5 9

AllowAccounts partition setting
by Marko Markoc 16 Oct '24

16 Oct '24

Hi All, At one point, when we were running slurm 22, we set up a few new partitions with `AllowAccounts` setting enabled. We limit access to these partitions only to a few accounts. We manually create these accounts and users using the `sacctmgr` command. At that time, everything was working. Only users in these accounts were able to run jobs in that partition. Few months ago we upgraded slurm to 23.02.7 but just a couple of days ago we noticed that anyone could submit jobs to these partitions. In addition to that, you can provide any value to the `--account` directive and slurm will accept and record it in the job accounting database. In our initial testing, you could only provide `--account` value that has been previously created. I'm still looking into this but I wanted to check if there was any change in version 23 that could have impacted this? I don't remember seeing anything in release notes. Thanks, Marko

6 8

Slurmctld process error 'double free or corruption' on RHEL 9 (Rocky Linux)
by William VINCENT 07 Oct '24

07 Oct '24

Hello I am writing to report an issue with the Slurmctld process on our RHEL 9 (Rocky Linux) . Twice in the past 5 days, the Slurmctld process has encountered an error that resulted in the service stopping. The error message displayed was "double free or corruption (out)". This error has caused significant disruption to our jobs, and we are concerned about its recurrence. We have tried troubleshooting the issue, but we have not been able to identify the root cause of the problem. We would appreciate any assistance or guidance you can provide to help us resolve this issue. Please let us know if you need any additional information or if there are any specific steps we should take to diagnose the problem further. Thank you for your attention to this matter. Best regards, _________________________ Jul 09 22:12:01 admin slurmctld[711010]: double free or corruption (fasttop) Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Main process exited, code=killed, status=6/ABRT Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Failed with result 'signal'. Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Consumed 11min 26.451s CPU time. ..... Jul 14 10:15:01 admin slurmctld[1633720]: double free or corruption (out) Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Main process exited, code=killed, status=6/ABRT Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Failed with result 'signal'. Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Consumed 7min 27.596s CPU time. _________________________ slurmctld -V slurm 22.05.9 ________________________ cat /etc/slurm/slurm.conf |grep -v '#' ClusterName=xxx SlurmctldHost=admin SlurmctldParameters=enable_configless SlurmUser=slurm AuthType=auth/munge CryptoType=crypto/munge SlurmctldPort=6817 StateSaveLocation=/var/spool/slurmctld SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmctldDebug=verbose DebugFlags=NO_CONF_HASH SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmdLogFile=/var/log/slurm/slurmd.log SlurmdDebug=verbose SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core,CR_LLN DefMemPerCPU=1024 MaxMemPerCPU=4096 GresTypes=gpu ProctrackType=proctrack/cgroup JobAcctGatherType=jobacct_gather/cgroup JobAcctGatherFrequency=15 JobCompType=jobcomp/none TaskPlugin=task/cgroup LaunchParameters=use_interactive_step AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=admin AccountingStoragePort=6819 AccountingStorageEnforce=associations AccountingStorageTRES=gres/gpu MailProg=/usr/bin/mailx EnforcePartLimits=YES MaxArraySize=200000 MaxJobCount=500000 MpiDefault=none ReturnToService=2 SwitchType=switch/none TmpFS=/tmpslurm/ UsePAM=1 InactiveLimit=0 KillWait=30 MessageTimeout=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 PriorityType=priority/multifactor PriorityFlags=FAIR_TREE,MAX_TRES PriorityDecayHalfLife=1-0 PriorityWeightFairshare=10000 NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=3500 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=2 Sockets=2 RealMemory=1700 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=1700 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=3500 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=r9nc-24-[1-12] NodeHostname=r9nc-24-[1-12] Sockets=2 CoresPerSocket=12 ThreadsPerCore=1 CPUs=24 RealMemory=180000 State=UNKNOWN NodeName=r9nc-48-[1-4] NodeHostname=r9nc-48-[1-4] Sockets=2 CoresPerSocket=24 ThreadsPerCore=1 CPUs=48 RealMemory=480000 State=UNKNOWN NodeName=r9ng-1080-[1-7] NodeHostname=r9ng-1080-[1-7] Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 CPUs=20 RealMemory=180000 State=UNKNOWN Gres=gpu:1080ti:4 NodeName=r9ng-1080-8 NodeHostname=r9ng-1080-8 Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 CPUs=20 RealMemory=176687 State=UNKNOWN Gres=gpu:1080ti:1 PartitionName=24CPUNodes Nodes=r9nc-24-[1-12] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=7500 DefMemPerCPU=7500 TRESBillingWeights="CPU=1.0,Mem=0.125G" Default=YES PartitionName=48CPUNodes Nodes=r9nc-48-[1-4] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=10000 DefMemPerCPU=8000 TRESBillingWeights="CPU=1.0,Mem=0.125G" PartitionName=GPUNodes Nodes=r9ng-1080-[1-7] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=9000 DefMemPerCPU=9000 PartitionName=GPUNodes1080-dev Nodes=r9ng-1080-8 State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=9000 DefMemPerCPU=9000 Hidden=Yes _________________________ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST 24CPUNodes* up infinite 12 idle r9nc-24-[1-12] 48CPUNodes up infinite 2 idle r9nc-48-[1-2] GPUNodes up infinite 4 idle r9ng-1080-[4-7] GPUNodes1080-dev up infinite 1 idle r9ng-1080-8 -- William VINCENT Administrateur systèmes et réseaux

4 15

Job Step State
by Emyr James 01 Oct '24

01 Oct '24

Dear all, I am working on a script to take completed job accounting data from the slurm accounting database and insert the equivalent data into a clickhouse table for fast reporting I can see that all the information is included in the cluster_job_table and cluster_job_step_table which seem to be joined on job_db_inx To get the cpu usage and peak memory usage etc. I can see that I need to parse the tres columns in the job steps. I couldn't find any column called MaxRSS in the database even though the sacct command prints this. I then found some data in tres_table and assume that sacct is using this. Please correct me if I'm wrong and if sacct is getting information from somwhere other than the accounting database? for the state column I get this... select state, count(*) as num from crg_step_table group by state order by num desc limit 10; +-------+--------+ | state | num | +-------+--------+ | 3 | 590635 | | 5 | 28345 | | 4 | 4401 | | 11 | 962 | | 1 | 8 | +-------+--------+ When I use sacct I see statuses seach as COMPLETED, OUT_OF_MEMORY etc. so there must be a mapping somewhere between these state ids and that text. Can someone prvide that mapping or point me to where it's defined in the database or in the code ? Many thanks, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

2 2

Can't schedule on cloud node: State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING
by Xaver Stiensmeier 20 Sep '24

20 Sep '24

Dear slurm-user list, I have a cloud node that is powered up and down on demand. Rarely it can happen that slurm's resumeTimeout is reached and the node is therefore powered down. We have set ReturnToService=2 in order to avoid the node being marked down, because the instance behind that node is created on demand and therefore after a failure nothing stops the system to start the node again as it is a different instance. I thought this would be enough, but apparently the node is still marked with "NOT_RESPONDING" which leads to slurm not trying to schedule on it. After a while NOT_RESPONDING is removed, but I would like to move it directly from within my fail script if possible so that the node can return to service immediately and not be blocked by "NOT_RESPONDING". Best regards, Xaver

2 3

Best practices for tracking jobs started across multiple clusters for accounting purposes.
by Di Bernardini, Fabio 02 Sep '24

02 Sep '24

I need to account for jobs composed of multiple jobs launched on multiple federated (and non-federated) clusters, which therefore have different job IDs. What are the best practices to prevent users from bypassing this tracking? NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico

3 4

QOS MaxTRESPU node=X intepretation
by David Magda 30 Aug '24

30 Aug '24

Hello, Have a question on how to interpret a Node=X MaxTRESPU value for a QOS: If (e.g.) X=4, and each node has (say) 64 CPUs or cores: if a particular job needs 32 cores, then would two jobs count as the equivalent of one node (2*32=64)? And if X=4 for the QOS, would that mean that eight of these jobs would be able to run ((4*64)/32=8)? Or does this TRES limit only get activated -N/--nodes is used? Don't see anything obvious: https://slurm.schedmd.com/resource_limits.html https://slurm.schedmd.com/sacctmgr.html https://slurm.schedmd.com/tres.html Thanks for any info. Regards, David

1 0

playing with --nodes=<size_string>
by Matteo Guglielmi 30 Aug '24

30 Aug '24

Hello, I have a cluster with four Intel nodes (node[01-04], Feature=intel) and four Amd nodes (node[05-08], Feature=amd). # job file #SBATCH --ntasks=3 #SBATCH --nodes=2,4 #SBATCH --constraint="[intel|amd]" env | grep SLURM # slurm.conf PartitionName=DEFAULT MinNodes=1 MaxNodes=UNLIMITED # log SLURM_JOB_USER=software SLURM_TASKS_PER_NODE=1(x3) SLURM_JOB_UID=1002 SLURM_TASK_PID=49987 SLURM_LOCALID=0 SLURM_SUBMIT_DIR=/home/software SLURMD_NODENAME=node01 SLURM_JOB_START_TIME=1724932865 SLURM_CLUSTER_NAME=cluster SLURM_JOB_END_TIME=1724933465 SLURM_CPUS_ON_NODE=1 SLURM_JOB_CPUS_PER_NODE=1(x3) SLURM_GTIDS=0 SLURM_JOB_PARTITION=nodes SLURM_JOB_NUM_NODES=3 SLURM_JOBID=26 SLURM_JOB_QOS=lprio SLURM_PROCID=0 SLURM_NTASKS=3 SLURM_TOPOLOGY_ADDR=node01 SLURM_TOPOLOGY_ADDR_PATTERN=node SLURM_MEM_PER_CPU=0 SLURM_NODELIST=node[01-03] SLURM_JOB_ACCOUNT=dalco SLURM_PRIO_PROCESS=0 SLURM_NPROCS=3 SLURM_NNODES=3 SLURM_SUBMIT_HOST=master SLURM_JOB_ID=26 SLURM_NODEID=0 SLURM_CONF=/etc/slurm/slurm.conf SLURM_JOB_NAME=mpijob SLURM_JOB_GID=1002 SLURM_JOB_NODELIST=node[01-03] <<<=== why three nodes? Shouldn't this still be two nodes? Thank you.

2 5

Print Slurm Stats on Login
by Paul Edmon 29 Aug '24

29 Aug '24

We are working to make our users more aware of their usage. One of the ideas we came up with was to having some basic usage stats printed at login (usage over past day, fairshare, job efficiency, etc). Does anyone have any scripts or methods that they use to do this? Before baking my own I was curious what other sites do and if they would be willing to share their scripts and methodology. -Paul Edmon-

10 20

Multiple Counts Question
by Matteo Guglielmi 29 Aug '24

29 Aug '24

Hello, Does anyone know why this is possible in slurm: --constraint="[rack1*2&rack2*4]" and this is not: --constraint="[rack1*2|rack2*4]" ? Thank you.

1 0

REST API - get_user_environment
by jpuerto＠psc.edu 29 Aug '24

29 Aug '24

In previous versions (v0.0.36) of the REST API the job submission endpoint had a field titled "get_user_environment"; however, it doesn't appear to exist in v0.0.40. Is there an equivalent parameter that should be used? What is the suggested approach for mimicking this behavior in v0.0.40? Best regards, Juan

5 13

ResumeAfterTime - Lacking Info
by Sid Young 28 Aug '24

28 Aug '24

G'Day all, Can anyone shed light on the parameter "Resume AfterTime" returned from the command "scontrol show node XXX" Can it be used to automatically resume a "Down"ed node? Sid

1 0

Invitation for Feedback on the Slurm-Lab Project
by Patrick Pun 28 Aug '24

28 Aug '24

Hi Slurm User Group, I’m excited to share a project I’ve been working on: Slurm-Lab. It’s a containerised, easy-to-deploy environment designed for learning, testing, and experimenting with Slurm. This project has allowed me and some of my friends to explore various Slurm features directly on our laptops, and I hope it can be just as useful to some of you. Highlights: • Quick Deployment: Setting up a Slurm environment is a breeze—just clone the project and pull the image to get started. • User-Friendly Interface: Interact with your sandbox environment seamlessly through JupyterHub in your browser. • Educational Focus: Explore different Slurm features with provided examples and interactive notebooks, perfect for both new and experienced users. Why I Need Your Feedback: As members of the Slurm community, your insights and experience are incredibly valuable. I’m seeking your feedback on how Slurm-Lab can better serve our community. Whether it’s feature suggestions, bug reports, or general comments. Please take a look at the project and share your thoughts. You can contribute by opening issues or submitting merge requests directly on GitLab. GitLab Repo: https://gitlab.com/CSniper/slurm-lab GitHub Mirror: https://github.com/csniper-patrick/slurm-lab Thank you for your time and for helping make Slurm-Lab a valuable resource for the entire community! Cheers, Patrick

1 0

Spread a multistep job across clusters
by Di Bernardini, Fabio 27 Aug '24

27 Aug '24

Hi everyone, for accounting reasons, I need to create only one job across two or more federated clusters with two or more srun steps. I'm trying with hetjobs but it's not clear to me from the documentation (https://slurm.schedmd.com/heterogeneous_jobs.html) if this is possible and how to do it. I'm trying with this script, but the steps are executed on only the first cluster. Can you tell me if there is a mistake in the hetjob or if it has to be done in another way? #!/bin/bash #SBATCH hetjob #SBATCH --clusters=cluster2 srun -v --het-group=0 hostname #SBATCH hetjob #SBATCH --clusters=cluster3 srun -v --het-group=1 hostname NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico

3 2

Slurm versions 24.05.3 and 23.11.10 are now available
by Marshall Garey 27 Aug '24

27 Aug '24

We are pleased to announce the availability of Slurm versions 24.05.3 and 23.11.10. Version 24.05.3 fixes a potential database problem when deleting a qos. This bug only existed in 24.05. Both versions have fixes for jobs potentially being stuck when using cloud nodes when some nodes are powered down, a regression in 23.11.9 and 24.05.2 that caused sattach to crash, and some other minor issues. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.05.3 > ========================== > -- data_parser/v0.0.40 - Added field descriptions > -- slurmrestd - Avoid creating new slurmdbd connection per request to > '* /slurm/slurmctld/*/*' endpoints. > -- Fix compilation issue with switch/hpe_slingshot plugin. > -- Fix gres per task allocation with threads-per-core. > -- data_parser/v0.0.41 - Added field descriptions > -- slurmrestd - Change back generated OpenAPI schema for > `DELETE /slurm/v0.0.40/jobs/` to RequestBody instead of using parameters > for request. slurmrestd will continue accept endpoint requests via > RequestBody or HTTP query. > -- topology/tree - Fix issues with switch distance optimization. > -- Fix potential segfault of secondary slurmctld when falling back to the > primary when running with a JobComp plugin. > -- Enable --json/--yaml=v0.0.39 options on client commands to dump data using > data_parser/v0.0.39 instead or outputting nothing. > -- switch/hpe_slingshot - Fix issue that could result in a 0 length state file. > -- Fix unnecessary message protocol downgrade for unregistered nodes. > -- Fix unnecessarily packing alias addrs when terminating jobs with a mix of > non-cloud/dynamic nodes and powered down cloud/dynamic nodes. > -- accounting_storage/mysql - Fix issue when deleting a qos that could remove > too many commas from the qos and/or delta_qos fields of the assoc table. > -- slurmctld - Fix memory leak when using RestrictedCoresPerGPU. > -- Fix allowing access to reservations without MaxStartDelay set. > -- Fix regression introduced in 24.05.0rc1 breaking srun --send-libs parsing. > -- Fix slurmd vsize memory leak when using job submission/allocation commands > that implicitly or explicitly use --get-user-env. > -- slurmd - Fix node going into invalid state when using CPUSpecList and > setting CPUs to the # of cores on a multithreaded node > -- Fix reboot asap nodes being considered in backfill after a restart. > -- Fix --clusters/-M queries for clusters outside of a federation when > fed_display is configured. > -- Fix scontrol allowing updating job with bad cpus-per-task value. > -- sattach - Fix regression from 24.05.2 security fix leading to crash. > -- mpi/pmix - Fix assertion when built under --enable-debug. > * Changes in Slurm 23.11.10 > =========================== > -- switch/hpe_slingshot - Fix issue that could result in a 0 length state file. > -- Fix unnecessary message protocol downgrade for unregistered nodes. > -- Fix unnecessarily packing alias addrs when terminating jobs with a mix of > non-cloud/dynamic nodes and powered down cloud/dynamic nodes. > -- Fix allowing access to reservations without MaxStartDelay set. > -- Fix scontrol allowing updating job with bad cpus-per-task value. > -- sattach - Fix regression from 23.11.9 security fix leading to crash.

1 0

Slurm management of Lenovo SD665 V3 dual-server trays?
by Ole Holm Nielsen 26 Aug '24

26 Aug '24

We're experimenting with ways to manage our new racks of Lenovo SD665 V3 dual-server trays with Direct Water Cooling (further information is in our Wiki page https://wiki.fysik.dtu.dk/ITwiki/Lenovo_SD665_V3/ ) Management problems arise because 2 servers share a tray with common power and water cooling. This wouldn't be so bad if it weren't for Lenovo's NVIDIA/Mellanox SharedIO Infiniband adapters, where the left-hand node's IB adapter is a client of the right-hand node's adapter. So we can't reboot or power down the right-hand node without killing any MPI jobs that happen to be using the left-hand node. My question is if other Slurm sites owning Lenovo dual-server trays with SharedIO Infiniband adapters have developed some clever ways of handling such node pairs a single entity somehow? Is there anything we should configure on the Slurm side to make such nodes easier to manage? Thanks for sharing any insights, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 5

How to select a container runtime system?
by wdennis＠nec-labs.com 23 Aug '24

23 Aug '24

We are getting a few calls to support container workloads on our Slurm cluster; I want to support these user's usecases, so am looking into it now. The problem for me is, I'm not super-familiar with container runtimes excepting (regular rootful) Docker... I see that any Slurm-compatible runtime must be "rootless", so that takes out the integration with regular Docker. How does one choose between the many options (from what I gather, Apptainer/Singularity, Podman, and rootless Docker seem to be the most widely-used.) We also have to support GPUs in these containers, but I assume that any of these can support that. Can anyone tell me what the most popular Slurm-compatible container runtime may be, and plusses/minuses of the three I mentioned above? Thanks!

2 1

Slurmdbd purge and reported downtime
by Davide DelVento 23 Aug '24

23 Aug '24

I am confused by the reported amount of Down and PLND Down by sreport. According to it, our cluster would have had a significant amount of downtime, which I know didn't happen (or, according to the documentation "time that slurmctld was not responding", see https://slurm.schedmd.com/sreport.html) Could it be my purge settings causing this problem? How can I check (maybe in some logs, maybe in the future) if actually slurmctld was not responding? The expected long-term numbers should be less than the ones reported for last month when we had an issue with a few nodes.... Thanks! [davide@login ~]$ grep Purge /opt/slurm/slurmdbd.conf #JobPurge=12 #StepPurge=1 PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/22 -------------------------------------------------------------------------------- Cluster Utilization 2022-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- --------------- --------------- -------------- ---------------- --------- ---------------- cluster cpu 19.50% 12.07% 3.92% 64.36% 0.15% 100.03% cluster mem 16.13% 13.17% 4.56% 66.13% 0.00% 99.99% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/23 -------------------------------------------------------------------------------- Cluster Utilization 2023-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- --------------- --------------- -------------- --------------- --------- ---------------- cluster cpu 28.74% 18.80% 6.44% 45.77% 0.24% 100.02% cluster mem 22.52% 20.54% 7.38% 49.55% 0.00% 99.98% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- -------------- --------------- -------------- --------------- -------- --------------- cluster cpu 29.92% 24.88% 17.73% 27.45% 0.02% 100.00% cluster mem 20.07% 28.60% 19.57% 31.76% 0.00% 100.00% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=8/8/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-08-08T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Planned Reported --------- -------------- ------------- ------------ -------- -------------- -------- -------------- cluster cpu 15.96% 2.53% 0.00% 81.51% 0.00% 100.00% cluster mem 9.18% 2.22% 0.00% 88.60% 0.00% 100.00% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=7/7/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-07-07T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Planned Reported --------- -------------- -------------- ------------- -------- -------------- -------- -------------- cluster cpu 27.07% 2.57% 0.00% 70.34% 0.02% 100.00% cluster mem 17.35% 2.26% 0.00% 80.40% 0.00% 100.00%

2 2

trying to figure out how to troubleshoot cloud node resume/suspend
by Alex Chekholko 23 Aug '24

23 Aug '24

Hi all, I have a cloud cluster running in GCP that seems to have gotten stuck in a state where the slurmctld will not start/stop compute nodes, it just sits there with thousands of jobs in the queue and only a few compute nodes up and running (out of thousands). I can try to kick it by setting node states to drain/resume/power_up/power_down, whatever, nothing seems to happen. In slurmctld.log I get stuff like this: 2024-08-22 16:44:31,761 INFO: created 1 instances: nodes=hpcmarjola-computenodeset-384 but then in resume.log there is no corresponding entry One hypothesis I have is that it's taking like 30+mins to spin up a new node, but I know GCP doesn't take that long to create the VM, so how can I troubleshoot more what the slurm resume script is doing? e.g. I saw in the log: [2024-08-22T16:39:57.151] powering up node hpcmarjola-computenodeset-7 [2024-08-22T16:59:10.371] Node hpcmarjola-computenodeset-7 now responding I tried setting SlurmctldDebug = verbose #instead of info, so it prints power / resume status messages to the log And I see things like this: [2024-08-23T16:40:29.006] node hpcmarjola-computenodeset-1068 not resumed by ResumeTimeout(600) - marking down and power_save [2024-08-23T16:40:29.006] requeue job JobId=184535_503(185038) due to failure of node hpcmarjola-computenodeset-1068 [2024-08-23T16:40:29.006] Requeuing JobId=184535_503(185038) [2024-08-23T16:40:29.007] POWER: power_save: waking nodes hpcmarjola-computenodeset-[2154-2156] [2024-08-23T16:40:29.007] POWER: power_save: handle failed nodes hpcmarjola-computenodeset-[1065-1068] I'll try bumping this value but that seems crazy high: ResumeTimeout=2400 slurm 23.11.7 I read through these pages: https://slurm.schedmd.com/power_save.html my config files (generated by gcp hpc toolkit scripts): topology.conf: SwitchName=nodeset-root Switches=computenodeset SwitchName=computenodeset Nodes=hpcmarjola-computenodeset-[0-4999] cloud.conf: PrivateData=cloud LaunchParameters=enable_nss_slurm,use_interactive_step SlurmctldParameters=cloud_dns,enable_configless,idle_on_node_suspend SchedulerParameters=bf_continue,salloc_wait_nodes,ignore_prefer_validation SuspendProgram=/slurm/scripts/suspend.py ResumeProgram=/slurm/scripts/resume.py ResumeFailProgram=/slurm/scripts/suspend.py ResumeRate=2 ResumeTimeout=600 SuspendRate=2 SuspendTimeout=600 TreeWidth=16 TopologyPlugin=topology/tree NodeSet=x-login Feature=x-login PartitionName=x-login Nodes=x-login State=UP DefMemPerCPU=1 Hidden=YES RootOnly=YES NodeName=DEFAULT State=UNKNOWN RealMemory=126832 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=1 CPUs=8 NodeName=hpcmarjola-computenodeset-[0-4999] State=CLOUD NodeSet=computenodeset Nodes=hpcmarjola-computenodeset-[0-4999] PartitionName=compute Nodes=computenodeset State=UP DefMemPerCPU=15854 SuspendTime=600 Oversubscribe=Exclusive PowerDownOnIdle=NO Default=YES ResumeTimeout=600 SuspendTimeout=600 SuspendExcParts=x-login slurm.conf: # slurm.conf # https://slurm.schedmd.com/slurm.conf.html # https://slurm.schedmd.com/configurator.html ProctrackType=proctrack/cgroup SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmdPidFile=/var/run/slurm/slurmd.pid TaskPlugin=task/affinity,task/cgroup MaxNodeCount=64000 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory # # # LOGGING AND ACCOUNTING AccountingStoreFlags=job_comment JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=info SlurmdDebug=info DebugFlags=Power # # # TIMERS MessageTimeout=60 ################################################################################ # vvvvv WARNING: DO NOT MODIFY SECTION BELOW vvvvv # ################################################################################ SlurmctldHost=hpcmarjola-controller(internal-cloud-dns-redacted) AuthType=auth/munge AuthInfo=cred_expire=120 AuthAltTypes=auth/jwt CredType=cred/munge MpiDefault=none ReturnToService=2 SlurmctldPort=6820-6830 SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurm # # # LOGGING AND ACCOUNTING AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=hpcmarjola-controller ClusterName=hpcmarjola SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd-%n.log # # # GENERATED CLOUD CONFIGURATIONS include cloud.conf ################################################################################ # ^^^^^ WARNING: DO NOT MODIFY SECTION ABOVE ^^^^^ # ################################################################################ MaxArraySize = 200001 MaxJobCount = 200001

1 0

Manually initiating a DB purge
by Sid Young 22 Aug '24

22 Aug '24

G'Day all, I have 3 years worth of job records in the slurm DB and we do not have any need to actually track anything at this stage, I would like to keep 12months worth of jobs so I need to purge 2 years worth at some point.. Is there a command to issue via scontrol to kick off a SlurmPurge using the values in the slurmdbd.conf file (currently these are 1000days, so I can see a few dozen records disappear before I shorten it to 365days in incremental steps). Sid

2 4

active reservation lost when increasing the duration
by Danny Marc Rotscher 22 Aug '24

22 Aug '24

Dear All, we are running Slurm 23.11.8 and I loose an active reservation and the jobs running in that reservation, when I tried to increase the duration of that reservation. I only want to tell you, that you may not run into the same problem. Kind regards, Danny Rotscher

1 0

Slurm hanging behavior
by Richard Yang 20 Aug '24

20 Aug '24

Hello Slurm community, We are using slurm as the system to deploy training jobs on a large gpu cluster, but encounter a strange behavior. As new comers, we wonder if this is a known behavior. Below is some more info: * We are running a relatively older version 22.0.5 * At relatively higher load, we encountered hanging. It is particularly puzzling in the following sense: assume we have nodelist1 with 6 hosts and nodelist2 with 7 hosts. We run simple ‘hostname’. Deploying on nodelist1 alone or nodelusr2 alone will be fine, but with all 13 hosts, the debug messages show that the execution hang after showing that the last task done. It then hangs for exactly 180 seconds. Does anyone know the potential issue? We sure be happy to post more config details or debug messages. Thank you so much! Richard

1 0

Access to --constraint=<arg> in Lua cli_filter?
by Kevin Buckley 20 Aug '24

20 Aug '24

If I supply a --constraint=<arg> option to an sbatch/salloc/srun, does the arg appear inside any object that a Lua CLI Filter could access? I've tried this basic check if is_unset(options['constraint']) then slurm_errorf('constraint is unset ') end and seen that that object is, indeed, unset. Kevin

2 2

Unable to run sequential jobs simultaneously on the same node
by Arko Roy 20 Aug '24

20 Aug '24

I want to run 50 sequential jobs (essentially 50 copies of the same code with different input parameters) on a particular node. However, as soon as one of the jobs gets executed, the other 49 jobs get killed immediately with exit code 9. The jobs are not interacting and are strictly parallel. However, if the 50 jobs run on 50 different nodes, it runs successfully. Can anyone please help with possible fixes? I see a discussion almost along the similar lines in https://groups.google.com/g/slurm-users/c/I1T6GWcLjt4 But could not get the final solution. -- Arko Roy Assistant Professor School of Physical Sciences Indian Institute of Technology Mandi Kamand, Mandi Himachal Pradesh - 175 005, India Email: arko(a)iitmandi.ac.in Web: https://faculty.iitmandi.ac.in/~arko/

5 7

sreport syntax for TRES/GPU usage
by Robert Kudyba 16 Aug '24

16 Aug '24

In a 25 node heterogeneous cluster with 4 different types of GPUs, to get granular to see which GPUs were used most over a time period we have to set AccountingStorageTRES to something like: AccountingStorageTRES=gres/gpu,gres/gpu:rtx8000,gres/gpu:v100s,gres/gpu:a40,gres/gpu:a100 Unfortunately it's currently at: AccountingStorageTRES=gres/gpu At least all nodes have the same GPU within each node. What are some good options to sreport to get details on usage over a year, e.g., percentage of CPU vs GPU, which partitions/accounts used the most GPUs, etc. From this example: sreport -tminper -t Percent cluster utilization --tres="cpu,gres/gpu" start=2023-07-01 -------------------------------------------------------------------------------- Cluster Utilization 2023-07-01T00:00:00 - 2024-08-15T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Reserved Reported --------- -------------- ----------- ---------- -------- ----------- ---------- ----------- cluster cpu 43.81% 2.87% 0.00% 48.35% 4.97% 99.86% cluster gres/gpu 50.36% 3.59% 0.00% 46.05% 0.00% 100.38% Is that showing that 50% of all jobs were run with GPUs? How do we read the Idle column? Why does Reported show > 100% for gres?

1 0

Upgrade compute node to 24.05.2
by Sid Young 15 Aug '24

15 Aug '24

G'Day all, I've been upgrading cmy cluster from 20.11.0 in small steps to get to 24.05.2. Currently 1 have all nodes on 23.02.8, the controller on 24.05.2 and a single test node on 24.05.2. All are Centos 7.9 (upgrade to Oracle Linux 8.10 is Phase 2 of the upgrades). When I check the slurmd status on the test node I get: [root@hpc-dev-01 24.05.2]# systemctl status slurmd ● slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2024-08-15 10:45:15 AEST; 24s ago Main PID: 46391 (slurmd) Tasks: 1 Memory: 1.2M CGroup: /system.slice/slurmd.service └─46391 /usr/sbin/slurmd --systemd Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Considering each NUMA node as a socket Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Node reconfigured socket/core boundaries SocketsPerBoard=4:8(hw) CoresPerSocket=16:8(hw) Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Considering each NUMA node as a socket Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: slurmd version 24.05.2 started Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: *plugin_load_from_file: Incompatible Slurm plugin /usr/lib64/slurm/mpi_none.so version (23.02.8)* Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: error: Couldn't load specified plugin name for mpi/none: Incompatible plugin version Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: error: MPI: Cannot create context for mpi/none Aug 15 10:45:15 hpc-dev-01 systemd[1]: Started Slurm node daemon. Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: slurmd started on Thu, 15 Aug 2024 10:45:15 +1000 Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: CPUs=64 Boards=1 Sockets=8 Cores=8 Threads=1 Memory=257778 TmpDisk=15998 Uptime=2898769 CPUSpecL...ve=(null) Hint: Some lines were ellipsized, use -l to show in full. [root@hpc-dev-01 24.05.2]# We don't use MPI (life science workloads)... should I remove the file? If it is version 23.02.8 then doesn't 24.05.2 have that plugin built in? There are no references to mpi i the slurm.conf file Sid

2 1

Seeking Commercial SLURM Subscription Provider
by John Joseph 13 Aug '24

13 Aug '24

Dear All, Good morning. We successfully implemented a 4-node SLURM cluster with shared storage using GlusterFS and were able to run COMSOL programs on it. After this learning experience, we've determined that it would be beneficial to switch to a commercial SLURM subscription for better support. We are currently seeking a solution provider who can offer support based on a commercial subscription. I would like to reach out to the group for recommendations or advice on how we can avail these services commercially. Thank you.Joseph John

2 1

Annoying canonical question about converting SLURM_JOB_NODELIST to a host list for mpirun
by Jeffrey Layton 12 Aug '24

12 Aug '24

Good afternoon, I know this question has been asked a million times, but what is the canonical way to convert the list of nodes for a job that is container in a Slurm variable, I use SLURM_JOB_NODELIST, to a host list appropriate for mpirun in OpenMPI (perhaps MPICH as well)? Before anyone says, compile OpenMPI with Slurm, I can't change the Slurm installation. I have a script that does the conversion on a single node, but when I try a cluster that does not include the single node, I get an error: scontrol: error: host list is empty The line in the script corresponding to this is, list=$(scontrol show hostname $SLURM_NODELIST) I've tried using the env variable SLURM_JOB_NODELIST and I get the same error message. Thanks! Jeff

3 8

FairShare if there's only one account?
by Drucker, Daniel 10 Aug '24

10 Aug '24

Simple question: Does FairShare still work if every user is under one account? E.g.: $ sacctmgr show assoc format=Account,User Account User ---------- ---------- root root root mic mic asmith mic bsmith mic csmith mic djones mic ejones mic frubble Will it divide time up fairly between the users? I have: PriorityType=priority/multifactor PriorityFavorSmall=YES PriorityWeightAge=50000 PriorityWeightFairshare=100000 PriorityWeightJobSize=0 PriorityWeightQOS=0 In 21.08.8. -- Daniel M. Drucker, Ph.D. Director of IT, MGB Imaging at Belmont McLean Hospital, a Harvard Medical School Affiliate The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

5 31

Jobs distribution over CPUs
by Rafał Lalik 09 Aug '24

09 Aug '24

Hi, I have a very simple computing farm on a single PC with AMD Ryzen 7950X (2x16 cores). I have configured my slurm to use up to 25 CPUs: NodeName=palmer CPUs=25 RealMemory=40000 State=UNKNOWN # Boards=1 SocketsPerBoard=1 CoresPerSocket=16 ThreadsPerCore=2 PartitionName=main Nodes=ALL Default=YES MaxTime=INFINITE State=UP But using htop I see that with all 25 jobs running I use max 16 cores. It seems that like * 6 jobs are using 100% of CPU * 20 jobs use 50% CPU each I use slurm 24.05.2, but I am pretty sure in the past when using I think one of 22.x version, the jobs distribution was like 25 CPUs for 25 jobs with 100% each. Is there anything I can do in the configuration of my farm to have better utilisation of CPU cores? Right now like half of them is not really used. Regards, Rafał

1 0

The issue in the distribution of job
by Sundaram Kumaran 09 Aug '24

09 Aug '24

Dear All, May I have your suggestion in my issue facing, While the job is launched using "salloc -N4--mem 4000 -p active" I find the job is running in the one compute node and the other 3 machines are free, I don`t find the job is distributed evenly, May I have your suggestion, I do squeue /scontrol to find the job distribution and it displays the 4 machines but when I check on the respective machines I don`t find the job running only one machine takes the whole node, Is there any issue in my conf file or what needs to be done, May I have your suggestion pls. FYI, salloc -N4 --mem 4000 -p active [cid:image001.png@01DAE8B4.215FBA50] While using TOP, I find only Debussy is used heavily, I don`t find my job is evenly distributed, May I have your guidance pls. [cid:image002.jpg@01DAEA98.2573DE10] Regards, KumaranS This e-mail and any attachments are only for the use of the intended recipient and may contain material that is confidential, privileged and/or protected by the Official Secrets Act. If you are not the intended recipient, please delete it or notify the sender immediately. Please do not copy or use it for any purpose or disclose the contents to any other person.

2 1

Optimizing CPU socket affinities and NVLink
by Matthew R. Baney 08 Aug '24

08 Aug '24

Hello, I've recently adopted setting AutoDetect=nvml in our GPU nodes' gres.conf files to automatically populate Cores and Links for GPUs, which has been working well. I'm now wondering if I can prioritize having single GPU jobs scheduled on NVLink pairs (these are PCIe A6000s) where one of the GPUs in the pair is already running a single GPU job, assuming the CPU socket with affinity has enough cores to handle the job. We have some users wanting to run single GPU jobs and others wanting to run dual GPU jobs, both on the same nodes, so we would prefer not to configure each NVLink pair as a single GRES, for better job throughput. As is, I've observed that for a node with at least 4 GPUs and 2 sockets (one NVLink pair per socket), Slurm will prioritize evening out core allocation between the sockets. Once the second single GPU job is submitted, one GPU in each NVLink pair is taken up and a subsequent dual GPU job can still run, but doesn't have access to an NVLink pair. We've also got a few nodes where single GPUs have failed, resulting in some NVLink'd pairs and usually a single non-NVLink'd GPU (3 or 7 total GPUs). It'd be ideal if single GPU jobs also got prioritized for scheduling on the non-NVLink'd GPU in this case. Is this possible? All the best, Matthew -- Matthew Baney Assistant Director of Computational Systems mbaney(a)umd.edu | (301) 405-6756 University of Maryland Institute for Advanced Computer Studies 3154 Brendan Iribe Center 8125 Paint Branch Dr. College Park, MD 20742

1 0

Integrate podman to run containers
by amin.raeiszadeh＠gmail.com 08 Aug '24

08 Aug '24

Hello, I have a test cluster consist of two nodes, one as controller and the other as compute node. I followed all the steps from slurm documentation and I want to run jobs as containers but I get the following error when running podman run hello-world on controller node: time="2024-08-06T12:02:54+02:00" level=warning msg="freezer not supported: openat2 /sys/fs/cgroup/system.slice/slurmstepd.scope/job_332/step_0/user/arlvm6.ara.332.0.0/cgroup.freeze: no such file or directory" srun: error: arlvm6: task 0: Exited with exit code 1 time="2024-08-06T12:02:54+02:00" level=warning msg="lstat /sys/fs/cgroup/system.slice/slurmstepd.scope/job_332/step_0/user/arlvm6.ara.332.0.0: no such file or directory" time="2024-08-06T12:02:54+02:00" level=error msg="runc run failed: unable to start container process: unable to apply cgroup configuration: rootless needs no limits + no cgrouppath when no permission is granted for cgroups: mkdir /sys/fs/cgroup/system.slice/slurmstepd.scope/job_332/step_0/user/arlvm6.ara.332.0.0: permission denied" As I tracked on the compute node this path exists /sys/fs/cgroup/system.slice/slurmstepd.scope/ but it looks that could not create the job_332/step_0/user/arlvm6.ara.332.0.0 . The cgroup.conf: CgroupPlugin=cgroup/v2 ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes It is the oci.conf: EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" RunTimeQuery="runc --rootless=true --root=/run/user/1223609544/ state %n.%u.%j.%s.%t" RunTimeKill="runc --rootless=true --root=/run/user/1223609544/ kill -a %n.%u.%j.%s.%t SIGKILL" RunTimeDelete="runc --rootless=true --root=/run/user/1223609544/ delete --force %n.%u.%j.%s.%t" RunTimeRun="runc --rootless=true --root=/run/user/1223609544/ run %n.%u.%j.%s.%t -b %b" As you see I changed the kill command a bit because without SIGKILL param it could not kill the containers. I test again the oci run time on both controller and compute nodes and I think might be helpful to mention two points: the delete command will not work because if you kill the container then there is no resource to be deleted at least in my tests. there is no pause and resume in oci.conf but I test them and got the same error for freezer support and cgroup permissions.

1 0

Find out submit host of past job?
by Steffen Grunewald 07 Aug '24

07 Aug '24

Hello everyone, I've grepped the manual pages and crawled the 'net, but couldn't find any answer to the following problem: I can see that the ctld keeps a record of it below /var/spool/slurm - as long as the job is running or waiting (and shown by "squeue") - and that this stores the environment that contains SLURM_SUBMIT_HOST - but this information seems to be lost when the job finishes. Is there a way to find out what the value of SLURM_SUBMIT_HOST was? I'd be interested in a few more env variables, but this one should be sufficient for a start... Is "sacct" just lacking a job field, or is this info indeed dropped and not stored in the DB? Thanks, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

6 9

"error: slurm_auth_get_host: Lookup failed: Unknown host" in slurmctld.log
by Chris Taylor 06 Aug '24

06 Aug '24

I keep getting this logged on my Slurm control host: [2024-08-06T20:31:40.196] error: slurm_auth_get_host: Lookup failed: Unknown host I don't see an identifiable pattern and I'm not sure how to troubleshoot. The jobs being submitted at or around that time seem fine and nobody's complained about their jobs not running, so I can't track down a bad or nonexistent DNS record. Chris

1 0

With slurm, how to allocate a whole node for a single multi-threaded process?
by Henrique Almeida 06 Aug '24

06 Aug '24

Hello, everyone, with slurm, how to allocate a whole node for a single multi-threaded process? https://stackoverflow.com/questions/78818547/with-slurm-how-to-allocate-a-w… -- Henrique Dante de Almeida hdante(a)gmail.com

8 16

LRMS error: (-1) Job missing from SLURM."
by Felix 06 Aug '24

06 Aug '24

Hello at site RO-14-ITIM, after a power failure I get the following problem 2024-08-06 15:53:04 Finished - job id: c9INDmclYv5ngvuSSqSAreymYz3jwmOETUEmV71LDmABFKDm7KNpMn, unix user: 1900:1900, name: "org.nordugrid.ARC-CE-result-ops", owner: "/dc=eu/dc=egi/c=hr/o=robots/o=srce/cn=robot:argo-egi@cro-ngi.hr", lrms: SLURM, queue: debug, lrmsid: 274399, failure: "LRMS error: (-1) Job missing from SLURM." 2024-08-06 15:53:04 Finished - job id: tjJNDmclYv5ngvuSSqSAreymYz3jwmOETUEmd71LDmABFKDmePf7To, unix user: 1900:1900, name: "org.nordugrid.ARC-CE-result-ops", owner: "/dc=eu/dc=egi/c=hr/o=robots/o=srce/cn=robot:argo-egi@cro-ngi.hr", lrms: SLURM, queue: debug, lrmsid: 274400, failure: "LRMS error: (-1) Job missing from SLURM." 2024-08-06 15:53:04 Finished - job id: kiJNDmclYv5ngvuSSqSAreymYz3jwmOETUEml71LDmABFKDmCmwifm, unix user: 1900:1900, name: "org.nordugrid.ARC-CE-result-ops", owner: "/dc=eu/dc=egi/c=hr/o=robots/o=srce/cn=robot:argo-egi@cro-ngi.hr", lrms: SLURM, queue: debug, lrmsid: 274398, failure: "LRMS error: (-1) Job missing from SLURM." The jobs can not be seen in sinfo or squeue And indication on how where to look up the problem? Thank you Felix -- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

2 1

when running `salloc --gres=gpu:1` should I see all gpus in nvidia-smi ?
by Oren 05 Aug '24

05 Aug '24

Hello, When I am running this command: `salloc --nodelist=gpu03 -p A4500_Features --gres=gpu:1` and then automatically ssh to the job, what should I see when I run nvidia-smi? All the GPUs in the host or just a single one? Thanks

2 2

_refresh_assoc_mgr_qos_list: no new list given back keeping cached one
by joao.damas＠syngenta.com 05 Aug '24

05 Aug '24

Hi all, We are doing a simple setup for a Slurm cluster (version 23.11.6). We follow the documentation and we are trying a setup still without accounting or slurmdbd. The slurm.conf is really simple: ``` ClusterName=Develop SlurmctldHost=head # Slurm configuration AuthType=auth/munge CryptoType=crypto/munge SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdLogFile=/var/log/slurm/slurmd.log SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurmctld # Nodes NodeName=worker1 CoresPerSocket=2 Sockets=1 ThreadsPerCore=1 NodeName=worker2 CoresPerSocket=2 Sockets=1 ThreadsPerCore=1 # Partitions PartitionName=develop Default=YES MaxTime=UNLIMITED Nodes="worker1,worker2" ``` When running a simple `srun sleep 10`, all works well and the log file shows: [2024-05-15T12:34:12.741] sched: _slurm_rpc_allocate_resources JobId=1 NodeList=worker1 usec=549 [2024-05-15T12:34:22.775] _job_complete: JobId=1 WEXITSTATUS 0 [2024-05-15T12:34:22.775] _job_complete: JobId=1 done But when creating a scrip with the same sleep command, and submiting using `sbatch test.sh`, the log shows: [2024-05-15T12:35:39.916] _slurm_rpc_submit_batch_job: JobId=2 InitPrio=1 usec=368 [2024-05-15T12:35:40.000] error: _refresh_assoc_mgr_qos_list: no new list given back keeping cached one. [2024-05-15T12:35:40.000] sched: JobId=2 has invalid account [2024-05-15T12:35:40.145] sched/backfill: _start_job: Started JobId=2 in develop on worker1 [2024-05-15T12:35:50.172] _job_complete: JobId=2 WEXITSTATUS 0 [2024-05-15T12:35:50.172] _job_complete: JobId=2 done We have the same account with the UID and GID, as said in the documentation. Looking at the function that seems to spit out that error (https://github.com/SchedMD/slurm/blob/e9f28ede27795f525e62f998cb2d40931d884…), it appears like there should be some accounting setup? We do not have slurmdbd setup and the documentation states we should test basic functionality before implementing that daemon. Any tips? Thanks in advance. João

4 3

problem with squeue --json with version 24.05.1
by Markus Köberl 05 Aug '24

05 Aug '24

$ squeue --version slurm 24.05.1 $ squeue --json malloc(): invalid size (unsorted) Aborted forcing an older data_parser version works: $ squeue --json=v0.0.40 regards Markus Köberl -- Markus Koeberl Graz University of Technology Signal Processing and Speech Communication Laboratory E-mail: markus.koeberl(a)tugraz.at

4 5

Upgrade node while jobs running
by Sid Young 03 Aug '24

03 Aug '24

G'day all, I've been waiting for node to become idle before upgrading them however some jobs take a long time. If I try to remove all the packages I assume that kills the slurmstep program and with it the job. Sid

5 5

slumrestd 24.05.1: crashes when GET on /slurm/v0.0.41/nodes : unsorted double linked list corrupted
by Josef Dvořáček 01 Aug '24

01 Aug '24

Isn't this failure familiar to anyone? When I ask API endpoint "localhost:6820/slurm/v0.0.41/jobs", slurmrestd segrafults with unsorted double linked list corrupted. Anyone using this API endpoint without segfaulting? I do the get using curl: curl --header X-SLURM-USER-NAME:root --header X-SLURM-USER-TOKEN:$SLURM_JWT -G localhost:6820/slurm/v0.0.41/jobs In comparison, curl --header X-SLURM-USER-NAME:root --header X-SLURM-USER-TOKEN:$SLURM_JWT -G localhost:6820/slurm/v0.0.41/nodes Works well. josef čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: _on_url: [[localhost]:52909] url path: /slurm/v0.0.41/jobs query: (null) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: operations_router: [[localhost]:52909] GET /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: slurmrestd: operations_router: [[localhost]:52909] GET /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:52909] attempting user_name root token authentication pass through čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:52909] attempting user_name root token authentication pass through čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi.json"](0, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi.yaml"](1, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi"](2, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=2 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi","v3"](3, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match shares to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed shares čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","shares"](4, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match reconfigure to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed reconfigure čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","reconfigure"](5, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match diag to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed diag čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","diag"](6, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match ping to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed ping čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","ping"](7, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match licenses to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed licenses čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","licenses"](8, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: method skip for ["slurm","v0.0.41","job","submit"](9, GET != POST) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","job","submit"](9, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: method skip for ["slurm","v0.0.41","job","allocate"](10, GET != POST) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","job","allocate"](10, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match jobs to jobs: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match successful for ["slurm","v0.0.41","jobs"](11, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: operations_router: [[localhost]:52909] found callback handler: (0x0) callback_tag=0 path=/slurm/v0.0.41/jobs parser=data_parser/v0.0.41 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] did not provide a known content type header. Assuming URL encoded. čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _parse_http_accept_entry: found */* with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] accepts */* with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] found accepts */*=application/json with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug3: _resolve_mime: [[localhost]:52909] mime read: application/x-www-form-urlencoded write: application/json čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug3: _call_handler: [[localhost]:52909] BEGIN: calling ctxt handler: 0x7F9C9D294A36[0] for path: /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: wrap_openapi_ctxt_callback: [[localhost]:52909] GET using data_parser/v0.0.41 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x1 from 0x408376 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x408376 from 0x1 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: accounting_storage/slurmdbd: _connect_dbd_conn: Sent PersistInit msg čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x1 from 0x408376 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x408376 from 0x1 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: malloc(): unsorted double linked list corrupted čec 24 14:37:55 slurmserver2.koios.lan systemd[1]: Started Process Core Dump (PID 1502951/UID 0).

2 3

2024

slurm-users August 2024 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users August 2024